Sound operated menu method and apparatus

ABSTRACT

The apparatus and method of the invention relate to data entry and menu selection. Applications include: (a) data entry for ideographic languages, including Chinese, Japanese and Korean; (b) fast food ordering; (c) correction of documents generated by optical character recognition; and (d) computer access and speech synthesis by persons temporarily or permanently lacking normal motor capabilities. In a preferred embodiment, each option of a menu is associated respectively with a selectable region displayed adjacent an edge of a display, forming a perimeter menu and leaving a region in the center of the perimeter menu for the output of an application program. Selectable regions may be on the display, outside the display, or both. A menu option may be selected by clicking on the associated selectable region, by dwelling on it for a selection threshold period or by a cursor path toward the selectable region, or by a combination thereof Remaining dwell time required to select a selectable region is preferably indicated by the brightness of the selectable region. Submenus of a perimeter menu may also be perimeter menus and the location of a submenu option may be foretold by the appearance of its parent menu option. Menu options may be ideographs sharing a sound, a structure or another characteristic. Ideographs, which may be homophones of one another, may be associated with colored indicating regions and selection of an ideograph may be made by speaking the name of the associated color.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation-in-part application of pending priorinternational application Number PCT/US95/03591 filed Mar, 27, 1995,International Publication Date Oct, 3, 1996, International PublicationNo. WO 96/30822, entitled "Method of and Apparatus for Data Entry" whichdesignated the United States, as amended Apr 25, 1995 and May 26, 1995.

FIELD OF THE INVENTION

The present invention relates generally to interactive display terminalsand interactive display methods, and more particularly, to interactivedisplay terminals and interactive display methods for use by personstemporarily or permanently lacking normal motor capabilities. It alsorelates to systems and methods for the assessment of the motorcapabilities of persons lacking normal motor capabilities. It furtherrelates to interactive display terminals and interactive display methodsfor use in speech synthesis for persons having impaired speech. It alsorelates to systems and methods for the control of devices, includingappliances, by persons lacking normal motor capabilities. It furtherrelates to interactive display terminals and interactive display methodsfor selecting one menu option from a menu. It further relates to systemsand methods utilizing sound recognition for selecting a menu option froma menu. It further relates to data and order entry systems including,and data and order entry methods utilizing, an interactive displayterminal. It also relates to interactive display terminals andinteractive display methods for displaying and selecting ideographiccharacters, such as are used in the Chinese, Japanese and Koreanlanguages. It also relates to interactive display terminals andinteractive display methods for producing an indication of progresstoward and/or away from selection of a menu option.

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the World IntellectualProperty Organization patent file or records, but otherwise reserves allcopyright rights whatsoever.

BACKGROUND ART

Many persons suffer from various neurogenic muscular disorders, such asCerebral Palsy ("CP"), Traumatic Brain Injury, Spinal Cord Injury,Muscular Dystrophy, Amyotrophic Lateral Sclerosis and MultipleSclerosis. These conditions can result in a reduced ability tovoluntarily control or prevent the movement of parts of the body,including the head, limbs and digits, muscle stiffness, weakness,limited range of motion, abnormal posture, involuntary muscle tremors,involuntary muscle activity causing involuntary motion, impaired abilityto voluntarily stop motion, impaired ability to coordinate muscleactivity, and/or impaired ability to sense the position of a part of thebody. Any one of these symptoms may impair an affected individual's finemotor control. Moreover, while some individuals affected by aneuromuscular disorder may be able to exercise fine motor control withenormous effort, the struggle to do so often fatigues the individual,limiting the period of time the individual is capable or comfortableperforming the fine motor control task.

Neuromuscular disorders are often systemic in effect, impairing anindividual's ability to operate prosthetic devices, such as awheelchair, and to perform the activities of daily life, such asspeaking, walking and operating household appliances. Speech isfrequently affected since the mechanics of producing speech requirecoordination of many muscle groups--the muscles of the diaphragm whichpush air over the vocal cords, the muscles of the larynx, jaws, tongueand lips. The inability to use or coordinate these muscle groups mayresult in impaired speech. Depending upon the degree of impairment,speech may be totally absent, present but impaired to the point ofunintelligibility, or intelligible on the whole but with occasionalunintelligible words. The ability to walk is often affected sincewalking requires coordination and voluntary control of many musclegroups. Furthermore, impaired fine motor control may prevent or impedean individuals from effectively operating household appliances orcomputer input devices.

Devices are available which produce speech, control appliances andfacilitate computer access for some persons having neuromusculardisorders ("NMD operators"). Devices which produce speech forindividuals whose own speech is impaired, called Augmentative andAlternative Communication ("AAC") devices, allow the operator to selectwords or phrases by spelling the words, by specifying an abbreviationfor the phrase or by selecting a sequence of symbols, and then speak theselected words or phrases using an electronic speech synthesizer.However, due to the systemic nature of neuromuscular disorders, NMDoperators are often unable to efficiently use a standard keyboard andmouse. For example, an NMD operator who is unable to stop the movementof a limb with precision, when attempting to use a keyboard or mouse,may move his arm toward the target key or move the cursor toward thetarget object on the display but overshoot the target. If he hasinvoluntary tremors and cannot hold a limb still, then, when attemptingto use a keyboard, he may hit keys adjacent to his target key. If he hasinvoluntary motion moving left to right, then, when attempting to use akeyboard, he may have difficulty accessing an intended key on the rightside of the keyboard.

The benefits of interfacing an NMD operator to a general purposecomputer so that he may control the computer and devices attached to it("computer access") are both numerous, because many of the problemsfaced by the disabled are susceptible to a computer-driven solution, andprofound, because of the psychological deprivation occasioned by asevere physical disability. The benefits potentially obtained throughcomputer access for individuals affected by neuromuscular disordersinclude:

a. Speech synthesis. A computer connected to a speech synthesizerenables an NMD operator with impaired speech to direct the computer tospeak for him.

b. Device control. A user who is physically unable to operate ahousehold appliance, for example, a television, video cassette recorder,compact disc player, radio, alarm clock, telephone, light, thermostat,dimmer or power switch, may be able to control the appliance via acomputer equipped with an interface he can control.

c. Access to general purpose computer applications. NMD operators maymake use of the same general purpose computer application programs("applications") as able-bodied users, including applications for wordprocessing, database, computer-aided instruction, access to literatureaccessible via computer, spreadsheet, time management and computerutilities.

d. Enhanced self-esteem and peer approval. Adolescents with CP areobviously different from their peers. They are often surrounded bynon-normative assistive technology, e.g. wheelchair or walker, specialschool bus equipped with a chair lift, stair lift, standing aid, AACdevice, feeding apparatus, bath seat, toiletting apparatus, etc. Inaddition, they may drool, lacking the ability to coordinate lip closurewith swallowing. Nonetheless they are adolescents and need peer approvalto support them in their maturation from dependent children toindependent adults. Today, demonstrated facility with a computer is anemblem of intelligence among adolescents, so computer use providesadolescents the opportunity to prove their intelligence and thuspotentially rewards NMD operators with both self-esteem and peerapproval.

e. Privacy. Some severely disabled school-aged children require nearlyconstant physical assistance to transfer them to and from bed, to feedthem, help them with toiletting and personal hygiene, etc. Because theyare constantly attended, all their mistakes in class or when doinghomework are known to their attendant, often a family member. They donot have the opportunity to make mistakes in private. Computer use, ifit can be done without assistance, affords the NMD operator theopportunity to avoid the embarrassment of showing their failings totheir attendant.

f. Expanded personal interaction. Some severely disabled individuals,e.g. quadriplegics, are essentially incarcerated by their disability.They interact with their family or their caretakers, depending uponwhether they live at home or in an institution. Their circle of friendsis often very small. Using a computer and a modem, they can expand theircircle of friends to include the tens of thousands of people whoperiodically connect to worldwide electronic networks to tradeinformation on topics of mutual interest. Moreover, the interaction viapresent computer networks is mostly textual; there is no voice or visualinteraction between users. Since messages are customarily composed andread off-line to minimize connect time charges, no one even knows howlong it took the sender to enter the text. Electronic networks thusafford the disabled user an opportunity to relate to others on an equalfooting, not as a disabled person to his able-bodied peers, somethingmany NMD operators dearly want to do but were never able to.

NMD operators vary widely in their motor capabilities. Even individualshaving the same medical diagnosis may require completely differenttechnologies for computer access. Many NMD operators are able to use anoversize keyboard, a device having a pressure-sensitive surface dividedinto squares, each square associated with a letter of the alphabet. Thesquares may be sized to match the operator's abilities, but typicallyeach square is two inches on either side. NMD operators who are unableto efficiently use an oversize keyboard may use another conventionalcomputer access solution, called an "on-screen keyboard", which, asillustrated in FIG. 1, is a picture of keyboard drawn on a computerdisplay (1101). The operator selects a letter by pointing to thatletter's key image on the display with a pointing device ("pointer"),then indicating that he has reached his target either by operating aswitch, a process called selection by click, or by maintaining thelocation indicated by the pointer ("dwelling") on the key image for apredetermined period of time (the "selection threshold"), a processcalled selection by dwell. Switch operation includes, but is not limitedto, each of the following: opening the switch, closing the switch,opening the switch multiple times within a predetermined period, andclosing the switch multiple times within a predetermined period. Aprogram executing on the computer determines which letter the operatorhas selected and processes the letter or passes it to some otherapplication program which processes the letter as if it came from a truekeyboard.

Conventional pointing devices include a mouse, trackball, joystick(which may be integrated into a keyboard, e.g. TrackPoint II®), stylusand graphics tablet, lightpen, thumb wheel, touch screen, touch panel,head pointer, occulometer, intraoral pointer and eye tracker. They maybe active, e.g. a lightpen that emits an infrared beam, or passive, e.g.an eye tracker that uses images of an individual's eyes to determinewhere his eyes are focusing. Conventional switches include a button onthe mouse, a switch in the tip of the stylus actuated by pressure or therelease of pressure, a switch mounted on the user's wheelchair operatedby a turn of the head to or the switch below a keyboard key.

Dwell time may be continuous or discontinuous depending upon theoperator's motor capabilities. In continuous dwelling, if the operatormoves the cursor from one key image to another region of the display,the time accumulated on the key image is discarded so that if theoperator returns to that key image he must dwell on it for the fullselection threshold to select it. Discontinuous dwelling, by contrast,compensates for involuntary tremors which pull the operator off thedesired key image. Accumulated dwell time on a key image is remembered,so that on return to a key image, the operator need only dwell for aperiod equal to the difference between the selection threshold and thepreviously accumulated dwell time for that key image. Accumulated dwelltime is reset to zero for all key images following the selection of anyone key image. Conventional on-screen keyboards do not indicate to theoperator the dwell time associated with any key image.

There may be a single selection threshold period for all key images oreach key image may be associated with its own selection thresholdperiod. In the latter case, keys associated with shorter selectionthreshold periods are easier to select than keys associated with longerselection threshold periods.

As was mentioned earlier, computer access permits an NMD operator to runa variety of applications. One such application is speech synthesis. Ina computer-based speech synthesis system, a computer system displayingan on-screen keyboard is connected to a speech synthesizer. The operatorspells the desired word or words using the on-screen keyboard. These arethen spoken by the speech synthesizer. Another application of theon-screen keyboard is word processing. FIG. 2 illustrates an example ofa combined display of an on-screen keyboard and a word processingapplication program. The on-screen keyboard (0201) is shown on the lowerportion of a display connected to a computer system (not shown) whichalso executes the word processing application program whose output(0203) appears on the upper portion of the display. Letters selected bythe operator are input to the word processing application program.

Due to impaired fine motor control, many NMD operators have difficultyselecting a key image by click or by dwell and this difficulty increasesas the size of the key image decreases. FIG. 1 shows an on-screenkeyboard containing 81 total keys including 26 alphabetic keys, 10numeric keys, 12 function keys, 4 arrow keys and 29 special purposekeys. Drawing this many key images on a display restricts the size ofeach key image making each very difficult for many NMD operators toselect.

When a display is shared between application program output (0203) andan on-screen keyboard (0201), as is the display shown in FIG. 2, thesize of each key image must be reduced from its size in FIG. 1 to allowspace for the application program output. Thus, as more display space isallotted to application program output, the key images become moredifficult for an NMD operator to select.

Many NMD operators have difficulty using the conventional dwellselectable on-screen keyboard because they cannot maintain a steadypointer position. The body member with which they control the pointermay move slightly ("drift") when they want it to remain still. Oneapproach to this problem is a variation of the on-screen keyboard,depicted in FIGS. 3, 4 and 5 and called a quaternary on-screen keyboard("quaternary keyboard"). The quaternary keyboard provides for larger keyimages. The alphabet is divided into four groups of letters, eachdisplayed in one of the four quadrants (1302), (1304), (1306) and (1308)of the display, as shown in FIG. 3. The operator selects one of the fourgroups by, for example, pointing to and dwelling on one quadrant of thedisplay. The selected group is then exploded into four subgroups, eachdisplayed in one quadrant of the display, as shown in FIG. 4. Once morethe operator selects one of the four. The selected group is explodedinto four letters and each letter displayed in one quadrant of thedisplay, as shown in FIG. 5. The operator again selects one of the four.This letter is then input to an application program (not shown).

The quaternary keyboard illustrates the use of a menu hierarchy incomputer access. Each of the four groups of letters (1302), (1304),(1306) and (1308) is a menu option. Each of these menu options is itselfa menu which includes other menu options. A menu hierarchy exists if atleast one of a menu's menu options is itself a menu. Hereinafter, a menuaccessed from another menu may be called a submenu, and the options ofthe submenu may be called submenu options. If a menu hierarchy is narrowand deep, many selections are required to make the desired choice. If amenu hierarchy is broad and shallow, each layer is composed of many menuoptions.

The quaternary keyboard greatly expands the size of a single key imageand thus accommodates certain NMD operators with drift or involuntarytremors. The cost of this adjustment is high Instead of selecting aletter with one pointing motion and dwelling for one selectionthreshold, the quaternary keyboard requires three pointing motions anddwelling for three selection thresholds. Thus the operator'sproductivity is dramatically reduced from the standard on-screenkeyboard depicted in FIG. 1.

The computer access advantage gained from the quaternary keyboard isgreatest when the quaternary keyboard occupies the entire display. Inthis configuration the size of each of the four active display regionsis maximized, making them easier to hit and dwell on for the operator.However, this configuration allows no room on the display for the outputof the application program being run by the operator, the reason he issitting at the computer in the first place. This does not prevent theon-screen keyboard from passing letters to the application program sincean application program need not be visible to be active, but it doesprevent the operator from seeing what the application program has toshow him. The more of the application program output that is displayed,the smaller the on-screen keyboard, the smaller each active region ofthe on-screen keyboard and the more difficult access becomes. FIG. 6illustrates a display combining a quaternary keyboard and output fromtwo application programs.

Another conventional structure for selecting of a menu option from amenu is a pie menu A pie menu is an opaque region on a display dividedinto selectable slices, each slice associated with a menu option. Thepie menu suffers some of the drawbacks discussed above, particularlythat, while displayed the pie menu occupies more space than a linearmenu and obscures much of the output of the operator's applicationprogram. For illustrations and a discussion of pie menus, see Callahan,Jack et. al., "An Empirical Comparison of Pie vs. Linear Menus",Computer Science Technical Report Series, CS-TR-1919, University ofMaryland, College Park, Md., September 1987.

NMD operators who cannot effectively use a conventional keyboard or apointing device may use a computer access method called "joystickpatterns". FIG. 7 depicts a conventional joystick pattern device. Thedevice (1602) is connected to a joystick and to a computer. The operatorpushes the joystick to the top, bottom, left, right, top left corner,top right corner, lower left corner or lower right corner, closing oneof eight switch contacts within the joystick housing. That switchposition is then indicated on the display (1604). A sequence ofconsecutive of switch closures encodes a letter or other programmedoutput that the device (1602) displays on an LCD display (1606) andsends to the connected computer, simulating keyboard input.

The conventional joystick pattern device is ill-suited for many NMDoperators. The involuntary tremors common some neuromuscular disordersmay result in unintended switch closures. In addition, the device doesnot provide an indication that the operator is moving a body member inan unintended direction until switch closure occurs. For example, anoperator with CP who intends to move the joystick the right but actuallymoves it to the upper right receives no indication from the device,prior to switch closure, that he's not on target. Moreover, the devicerequires that the operator memorize the encoding of each letter or otheroutput since there's no indication on the display (1606) which sequenceencodes which letter. Further, the device provides no support for headpointing, although the head is often the best controlled part of an NMDoperator's body.

NMD operators who cannot effectively use either a conventional keyboardor a pointing device but can reliably actuate a switch may use acomputer access method called "scanning", which is subdivided by cursorcontrol technique into three types of scanning: automatic, directed andstep. In automatic scanning all the operators' options, for example, theletters of the alphabet, appear on either a static or dynamic display(depending upon the implementation), organized in rows and columns. Atthe scanning interval, usually about one second, a cursor moves from onerow to the next. When the cursor indicates the row containing the letterthe operator wants, he closes a switch. The machine now moves the cursorfrom one letter to the next within the selected row until the operatorcloses the switch again. The operator has now selected one letter. Indirected scanning, like automatic scanning, the cursor moves at thefrequency determined by the scanning interval, however, it moves onlywhen the switch is closed. To select an option, such as a row or aletter in a row, the operator opens the switch while the cursorindicates the desired option. In step scanning, the cursor moves witheach switch activation.

As one can well imagine, writing a sentence via any of these scanningtechniques is an extremely slow process, since selecting a single lettermay take many seconds.

Problems of computer access cascade and affect the quality of verbalinteractions between AAC device operators ("AAC operators") and others.People speak much faster than they type. Not surprisingly, operators whospeak with AAC devices, particularly NMD operators whose motor deficitsimpair their ability to use a keyboard, lag substantially in theirconversations. The slow pace of an AAC operator's word productiondisrupts normal verbal interaction. Speaking persons, accustomed or notto the AAC operator's slow rate, often lose patience in conversationswith AAC operators. They may prematurely terminate the conversation,read the AAC device display in an attempt to guess at the AAC operator'sintended utterance and so accelerate the interaction, lead the AACoperator, ask predominantly yes/no questions, change the topic ofconversation with little input from the AAC operator and otherwisedominate the interaction. The AAC operator often has difficultyparticipating as an equal partner in the conversation. He may be unableto change the topic, interject a humorous comment in a timely fashion orrespond to a question before the speaking person changes the topic. SlowAAC operators may be perceived as mentally slow. Thus the quality ofverbal interactions where one party uses an AAC device to speak dependssignificantly upon the AAC operator's rate of word production.

Increasing an operator's letter or menu option selection rateproportionately increases his word production rate and increases theoperator's productivity in data entry generally. Letter or menu optionselection time includes the time the operator requires (a) to comprehendthe menu options displayed, (b) to move the pointer to the desired menuoption on the display, and, in selection by dwell, (c) the selectionthreshold period, or, in selection by click, (c) the time required tooperate the switch. Decreasing any one of these increases the operator'sproductivity, assuming all other steps in the selection process areunaffected.

Personal interactions are composed of more than speech alone. People inconversation gesture to one another, use facial expressions, change theobject of their gaze and make non-speech utterances (e.g. "hmmm-mmmm")to bid for a turn to speak, to grant such a bid made by the other party,to request to continue speaking and to acknowledge, accept or disputewhat has been said. Ideally, the production of speech from an AACdevices does not distract the AAC operator from the personal interactionand subject matter of the conversation. This is possible if the operatorhabituates the AAC device access technique and menu structure, producingspeech without focusing on each step of the process, much as automobiledrivers habituate mechanical tasks, such as changing gears and switchingbetween foot pedals, and focus their attention on pedestrians or trafficlights while operating their vehicle.

Another consequence of personal interaction during conversation for anAAC device operator is that the operator needs a way to easily enableand disable the AAC device operator interface so that movement theoperator makes during personal interaction, for example, nodding hishead, is not interpreted by the AAC device.

As noted previously, neurogenic muscular disorders may impair theability of an individual to sense the position of a body member. An NMDoperator thus relies more than his able-bodied peer on the location of acursor or similar automated indication of body member position.Conventional access methods which use a pointer do not provideadditional feedback to the operator of the position of a body member.

Access methods which require the NMD operator to make the same movementfor most selections, such as single switch access, mouth sticking (theuse of a small rod held in the mouth and used to depress keys on akeyboard) and head sticking (the use of a rod mounted on the head andused to depress keys on a keyboard), may result in repetitive motioninjury, particularly after years of use.

The need for quick selection from a menu also arises from the use ofoptical character recognition ("OCR") systems which attempt to recognizegraphic symbols and words based on attributes for optical recognitionpurposes, for example, the appearance of graphic symbols, the ratio ofdark space to light space within part or all of a graphic symbol, theratio of dark space in one part of a graphic symbol to the dark space inanother part of the graphic symbol, and the derivative of darkness overthe scan of the graphic symbol. OCR systems convert the contents of atypewritten document into a computer encoding of the same. OCR systemsat times are unable to recognize a graphic symbol or word, or may err inselecting a graphic symbol or word from a plurality of candidates.Therefore, following optical character recognition, a human mayproofread and correct the computer encoded document. The proofreader mayindicate where an error or omission in the computer encoded documentoccurred and may select from a plurality of menu options, eachrepresenting a candidate graphic symbol or word.

There are several aspects of the invention, each addressing one or moreof the problems described above and/or one or more problems specificallyaddressed by that aspect of the invention. The objects, disclosure anddescription of each aspect is separately described below under one ofthe headings: (A) Perimeter Menu, (B) Confinement, (C) Dwell, (D) PathDirectness, (E) Intersection, (F) Alignment, (G) Length Order, (H)Location Indication, (I) Sound Match, and (J) Ideographic Languages,.Where there is background art applicable to an aspect in addition tothat described above, the additional background art is described below.

A & B. Perimeter Menu and Confinement

One object of the invention is to facilitate computer access by adisabled operator.

A further object of the Perimeter Menu and Confinement aspects of theinvention is to facilitate menu selection by an operator having impairedability to maintain a body member in a steady position.

Another object of the Perimeter Menu and Confinement aspects of theinvention is to facilitate menu selection by an operator having impairedability to stop motion.

Yet another object of the Perimeter Menu and Confinement aspects of theinvention is to simultaneously display an application program window anda computer access menu which does not obstruct the application programwindow.

Another object of the Perimeter Menu and Confinement aspects of theinvention is to allow an operator to enable and disable a menu.

A still further object of the Perimeter Menu and Confinement aspects ofthe invention is to synthesize speech for an operator having impairedspeech and impaired fine motor control.

Still another object of the Perimeter Menu and Confinement aspects ofthe invention is to facilitate device control for a disabled operator.

Another object of the Perimeter Menu and Confinement aspects of theinvention is to reduce the cognitive demand of speech synthesis for thedisabled.

A still further object of the Perimeter Menu and Confinement aspects ofthe invention is to enlarge the effective area of a selectable regionwithout concomitantly reducing the area available for informationdisplay.

Another object of the Perimeter Menu and Confinement aspects of theinvention is to speed data entry.

Yet another object of the Perimeter Menu and Confinement aspects of theinvention is to facilitate computer access for an operator havingimpaired ability to sense the position of a body member used forcomputer access.

Another object of the invention is to facilitate the correction oferrors or completion of omissions in a computer encoded documentproduced, at least in part, by optical character recognition.

C. Dwell

Conventional systems allowing selection by dwell do not provide anindication to the operator of either how much dwell time has beenaccumulated for any selectable region or how much more dwell time isrequired to select a selectable region. Consequently, an operator of aconventional system who is dwelling on a intended selectable region hasno indication, other than his estimation from prior use of the system,that he has nearly made his selection and can plan his next movement tothe next selectable region or that he has very nearly made his selectionand can begin moving to the next selectable region. Furthermore, anoperator of a conventional system who is dwelling on an unintendedselectable region, has no indication, other than his estimation fromprior use of the system, of how close he is to making an unintendedselection and thus how important it is to act quickly. Conventionalsystems using discontinuous dwell give no indication of the accumulateddwell time associated with a selectable region either when the operatordwells on that region or when the operator ceases dwelling on thatregion. Some disabled users can dwell relatively easily on theirintended targets for short periods of time, but have difficulty dwellingfor long periods. If such an operator knows that only a little moredwell time is needed he may be able to satisfy the dwell time requiredfor selection, without preparing himself to dwell for an extendedperiod.

Conventional menu-driven data entry and order entry systemsincorporating pointing at intended selections employ a two stepselection procedure. In the first step the operator indicates, with apointer, his intended selection. The system then provides feedback, forexample, by highlighting the indicated selection, showing whichselection the operator has indicated. In the second step, the operatorselects the indicated selection, for example, by operating a switch.Thus, conventional data entry and order entry systems are ill-suited tocircumstances where the operator cannot easily operate a switch whilemaintaining the pointer on the intended selection.

While the two step procedure is not complicated, many operators requiresome training to learn it, and, if they are infrequent users of thesystem, these operators may require refresher training. Simplifying theprocedure further would lessen the need for initial and refreshertraining.

One object of the invention is to facilitate the use of systems allowingselection by dwell.

Still another object of the invention is to facilitate device control bythe disabled.

A further object of the invention is to increase the independence of thedisabled.

Yet another object of the invention is to facilitate the use of a dataentry or order entry system by an intermittent operator.

Another object of the invention is to facilitate ordering by someoneseated in a vehicle.

D. Path Directness

The on-screen keyboard with dwell selectable key images is ill-suitedfor use by many NMD operators. Selection by dwell may fatigue NMDoperators or may require greater fine motor control than they bring tothis task. Operators with impaired ability to stop motion and thosehaving involuntary tremors have difficulty maintaining the locationindicated by a pointer on a key image for a period sufficient todistinguish intentional dwelling from unintentional dwelling.Consequently, some NMD operators who try to use on-screen keyboardsoften miss their target key images and/or accidentally select unintendedkey images. Following such an error, the operator must erase hiserroneous selection by selecting the backspace or undo key. As thenumber of erroneous selections increases, the operator's productivitydecreases markedly, since each error requires a correction in whichthere might be another error.

Conventional on-screen keyboards require the ability to select by dwellor by click and thus are limited to operators with these capabilities.Conventional on-screen keyboards do not utilize the relatively intactmotor capabilities of some NMD operators to compensate for impairedability to select by dwell or by click or to speed up the slow processof selecting by dwell. For example, while an NMD operator may overshoota key image, his directional control may be relatively intact.Conventional on-screen keyboards do not exploit this capability.

The dominant computer operating system for graphic applications ongeneral purpose computer systems today is the Windows® Operating System.Windows® assigns meaning to the cursor location. When the operator movesthe cursor on top of a menu item and clicks, Windows® interprets theaction as manifesting an intent to choose that menu item. The operator'spath to that menu item, whether direct or circuitous, is irrelevant.Operators who can move toward a target accurately but cannot maintainthe location indicated by a pointer on the target cannot effectively usestandard Windows® applications through the conventional interface tothese applications.

Often NMD operators cannot steady a pointer while operating a switch;the act of operating the switch triggers involuntary muscle activitypulling the cursor off target. For these operators, conventionalselection by click is not practicable. Conventional selection by dwellalso requires greater fine motor control than many NMD operators bringto this task. Operators with impaired ability to stop motion mayovershoot their intended target. Operators whose voluntary muscleactivity is accompanied by some involuntary muscle activity affectingtheir directional control often cannot point accurately. Operators withinvoluntary tremors often cannot maintain the location indicated by apointer on a key image. Consequently, NMD operators who try to useon-screen keyboards often miss their target key images and accidentallyselect unintended key images. Following such an error, the operator musterase his erroneous selection by selecting the backspace or undo key. Asthe number of erroneous selections increases, the operator'sproductivity decreases markedly, since each error requires a correctionin which there might be another error.

Measures of an individual's fine motor control assist a physician ortherapist in evaluating the effectiveness of a treatment program,including assistive technology, and in gauging the severity of adisability. Such measures help the physician or therapist in determiningwhat treatment course to pursue and whether the severity of a certaindisability justifies the risk of a particular treatment option, such asneurosurgery.

One object of the invention is to facilitate selection of an option froma menu.

Another object of the invention is to indicate to an operator moving acursor toward an option in a menu displayed by a computer system, whichoption the computer system believes the operator is moving toward.

Another object of the invention is to display a menu on a display sothat a large contiguous area on the display is not obstructed by themenu.

Still another object of the invention is to make use, in computeraccess, of relatively unimpaired directional control in persons havingimpaired fine motor control.

A further object of the invention is to indicate to an operator moving acursor toward a dwell-selectable option in a menu, a changed selectionthreshold of the dwell-selectable option.

Yet another object of the invention is to more efficiently select anoption from a menu on a display.

Another object of the invention is to speed up selection of an optionfrom a menu by an operator having impaired ability to operate a switchwhile the operator simultaneously keeps a cursor location within aregion on a display.

Another object of the invention is to help an operator with a disabilitycontrol a pointer.

Yet another object of the invention is to speed data entry by anindividual with a disability.

A still further object of the invention is to facilitate artificialspeech generation by a person having impaired speech due to a neurogenicmuscular disorder.

Yet another object of the invention is to facilitate device control by aperson having a neurogenic muscular disorder.

Another object of the invention is to measure an individual's ability tomove one of the individual's body members in a direct path from astarting position to an ending position.

Yet another object of the invention is to facilitate computer access foran individual who cannot stop movement cleanly.

E. Intersection

One object of the invention is to facilitate computer access by anoperator having impaired ability to maintain a body member in a steadyposition.

Another object of the invention is to facilitate the selection of adesired menu option by an operator having impaired fine motor control.

A further object of the invention is to synthesize speech for anoperator having impaired speech and impaired motor control.

Yet another object of the invention is to use an operator's directionalcontrol in computer access.

A still further object of the invention is, in selecting a menu optionfrom a menu of dwell-selectable menu options, to compensate for anoperator's impaired ability to maintain a body member in a steadyposition by using the operator's relatively intact motor capability .

F. Alignment

Conventional on-screen keyboards do not compensate for NMD operators'inability to stop motion. Suppose the operator has been fitted with ahead pointing device so that his head motion moves the cursor and thathe's using the quaternary keyboard shown in FIGS. 3, 4 and 5. Assumefurther that, as he attempts to point to the quadrant containing the "j"key image (1308) in FIG. 3, he is unable to stop on that quadrant andcontinues turning 20 more degrees to the left. There are two known waysof responding to this situation: (1) the cursor may continue to trackthe operator's motion and disappear from the display, leaving noindication to the operator of the location of the cursor andconsequently causing some operator disorientation, or (2) the cursor may"stick", i.e. remain confined to the display, at, for example, point(1312). Conventional on-screen keyboards respond in this way. Theoperator's line of sight is now 20 degrees to the left of the cursorlocation. After the dwell period, the quadrant (1308) is selected andFIG. 4 is displayed. The cursor hasn't moved. It is now at point (1320).Assume that again the operator attempts to point to the quadrantcontaining the "j" key image (1324) in FIG. 4. As the operator turns hishead to the right, the cursor immediately moves with him. Thus, theoperator's line of sight remains 20 degrees to the left of the cursorlocation as the cursor moves to the right across the display. Theoperator must watch the cursor out of his right eye. The problem isaggravated if either the operator cannot cleanly stop or if he drifts ashe dwells. Assume that while attempting to dwell on quadrant (1324) theoperator drifts 25 degrees past the bottom of the screen. His line ofsight is now 25 degrees below and 20 degrees to the left of the cursor.To correct this misalignment in the conventional quaternary keyboard,the operator must turn his head to the right, "stick" the cursor againstthe right edge of the display, and continue turning 20 degrees until hehas the cursor in his line of sight. Then he must lift his head until hesticks the cursor against the top edge and continue lifting 25 degreesmore. Alternatively, in this scenario, the operator could stick thecursor in the upper right corner of the display and simultaneouslyrotate his head up and to the right until he brought the cursor into hisline of sight.

Alignment is also a problematic for NMD operators who use a pointer,such as a mouse, with which the operator indicates by a location on asurface, e.g. a desk top, which corresponds to a desired location on thedisplay, and achieve alignment by removing the pointer from the surface,e.g. lifting the mouse, moving the mouse, then replacing it on thesurface. Due to impaired fine motor control, many NMD operators cannotremove a pointer from the surface and replace it on the surface at adesired location without unintentional movement or extraordinary effort.For these operators, alignment cannot be effectively achieved throughconventional means.

In summary, misalignment interferes with accurate pointing and theprocess of correcting for misalignment may result in the selection ofunintended key images.

One object of the invention is to allow an operator to align a pointerwith a location on a surface.

Another object of the invention is to indicate to an operator a locationon a surface with which he may align a pointer.

Still another object of the invention is to indicate to an operator whenhe may align a pointer with a location on a surface.

A further object of the invention is to allow an operator havingimpaired motor control to align a pointer with a cursor.

G. Length Order

As noted previously, one of the elements determining the menu optionselection time is the time the operator requires to comprehend the menuoptions displayed. This time may be reduced if the operator can limitthe number of menu options he searches in looking for his desired menuoption.

Conventional word prediction systems attempt to reduce this operatorsearch time. The operator of a conventional word prediction system may,for example, select the letter "p". The system displays some number, saysix, of the most frequently used words beginning with the letter "p".Conventionally these six words are displayed either in alphabetic orderor in order of frequency of use. Assuming the operator does not see hisdesired word on the display, he selects another letter, say "r". Thesystem then displays the six most frequently used words beginning withthe letters "pr".

Searching a displayed list of words in alphabetic order requires thatthe operator focus his attention on the selection task, as opposed tothe information content of the conversation or other task the operatoris engaged in. Further, determining whether a given word isalphabetically greater or lesser than a desired word takes substantialtime, slowing the selection process. An alphabetically ordered list isof limited use to an individual who has below normal spelling ability, afrequent problem among individuals with impaired speech. Ordering wordsby frequency of use often does not limit the number of words theoperator must search. The word at the bottom of the displayed list, forexample, the sixth most frequently used word beginning with the letters"pr" may be a very common word, even though it is less frequently usedthan the other five displayed words.

One object of the invention is to reduce the time an operator requiresto comprehend displayed menu options.

Another object of the invention is to reduce operator search time.

Still another object of the invention is to limit the number of menuoptions an operator searches for in looking for his desired menu option.

A further object of the invention is to speed data entry.

Yet another object of the invention is to increase productivity inspeech synthesis for an operator having impaired speech.

H. Location Indication

The difficulties experienced by NMD operators in pointing to relativelysmall selectable regions have already been described. One approach tothese difficulties is to enlarge the on-screen selectable region,illustrated by the quaternary expansion on-screen keyboard alreadydescribed. Another approach is the conventional eye gaze system for aspeech impaired individual, depicted in FIG. 8. The system consists of aplexiglass frame (6352) having a centrally located aperture (6354). Theeye gaze system is positioned between the speech impaired individual andperson with whom the speech impaired individual is communicating. Thereare eight groups of five squares each on the plexiglass frame. Eachsquare within each group of five squares is color coded, e.g. red, blue,green, yellow and clear, matching the color on each of the four cornersof the plexiglass frame. The clear square matches the aperture (6354).All squares are labeled with symbols representing items to becommunicated. These labels are not shown in FIG. 8. The person with whomthe speech impaired individual is communicating observes the eyes of thespeech impaired individual to determine the target of the speechimpaired individual's eye gaze. To communicate an item, the speechimpaired individual gazes first toward the one of the eight groups offive squares, indicating that he wants to communicate one of the symbolsin that group, and gazes second toward one of the four corners andaperture (6354) matching the color of the square labeled with the itemto be communicated in the previously indicated group.

Two types of selectable regions are conventionally used in a point andclick menu interface in a graphical user interface. The first, shown inFIG. 9, depicts a menu having three menu options, labeled "High","Medium" and "Low", each displayed on a display (4807), each associatedrespectively with selectable regions (4801), (4803) and (4805), and eachlocated adjacent the associated selectable region. FIG. 10 depicts amenu having the same three menu options, each displayed on a display(4807), each associated respectively with selectable regions (4901),(4903) and (4905), and each intersecting the associated selectableregion. In both these conventional menus, a menu option is selected bypointing to and clicking on the associated selectable region.

Conventional menu hierarchies in automated systems, built from menus ofthe type shown in FIG. 9 or FIG. 10, require that the operator proceedsequentially through the steps of searching menu options, selecting oneof them, and, assuming a menu option including a submenu was selected,searching the submenu options, and selecting one of them. Whereselection from menu hierarchies constitutes a substantial component ofthe operator's activities, the slowness of the selection processdiminishes productivity.

Locating selectable regions or parts thereof outside the display, inaccordance with the Perimeter Menu aspect of the invention, allows thelarge areas outside the display to be used, a major advantage foroperators having impaired fine motor control who are unable to maintaina pointer on a small selectable region while selecting by click or bydwell. However, if menu options are displayed on the display near theperimeter of display and near their associated selectable region, theoperator has an indication of the location of each selectable region butmay not be able to see all the displayed menu options in a glance.Because an operator usually searches a displayed menu for his intendedmenu option, placing the menu only near the perimeter of the display mayincrease menu search time, thus increasing menu option selection time.

One object of the invention is to indicate to an operator of a menusystem having selectable regions outside the display, the menu optionassociated with each selectable region and the location of eachselectable region.

Another object of the invention is to facilitate selection from a menuby an operator having impaired motor control.

Still another object of the invention is to speed selection of a menuoption from a menu and of a submenu option from a menu hierarchy.

A further object of the invention is to speed speech synthesis for aperson having impaired speech and impaired motor control.

I. Sound Match

Conventional speech recognition systems facilitate computer access forindividuals unable to use a standard keyboard whose speech is relativelyunimpaired, for example, an individual with quadriplegia, and hands-freecomputer access for able-bodied individuals. The operator of such aspeech recognition system reads a menu option out loud, for example,"open file", and the system, which includes sound receiving means, forexample, a microphone coupled to a sound board having a Digital SignalProcessor ("DSP"), receives the sound of the read menu option, digitizesthe sound of the read menu option, and then provides the digitized soundto another component of the speech recognition system, sound matchingmeans which includes an application program for matching the digitizedsound to one of a plurality of sounds, each representing respectivelythe sound of a spoken menu option. The system determines which soundbest matches the sound of the read menu option and selects the menuoption associated with this best matched sound.

Individuals whose speech is impaired are often unable to effectively useconventional speech recognition systems because they often cannotproduce a large distinct variety of sounds characteristic of phoneticlanguages. For example, such an individual may produce similar soundsfor the two consonants "t" and "d" so that these sound areindistinguishable to a conventional speech recognition system, or suchan individual may not be able to consistently produce soundsdistinguishable by a speech recognition system, resulting in falsematches. Other symptoms of impaired speech, for example, similaritiesamong certain phonemes and impaired ability to start or stop soundproduction appropriately, may substantially limit the variety of soundsdistinguishable to a conventional speech recognition system anindividual may consistently produce.

Conventional speech recognition systems provide limited capabilities inlanguages rich in homophones, for example, Chinese, because in suchlanguages, a distinct sound is often insufficient to specify a word, asis described in the Background Art section of the Ideographic Languagesaspect of the invention. The problem may be briefly illustrated by anexample. Suppose a Chinese data entry operator using a conventionalspeech recognition system speaks the phonetic unit "fu" with aparticular intonation. This distinct sound may well have over 15homophones. Although the operator could use the keyboard to select oneof these 15 homophones, this defeats the purpose of speech recognition,which is to facilitate hands-free computer access.

One object of the invention is to facilitate selection from a menu, and,in particular, from a menu of homophones.

Another object of the invention is to facilitate speech synthesis andvoice activated computer access by individuals with speech impairments.

Still another object of the invention is to speed data entry inIdeographic Languages.

J. Ideographic Language

The use of ideographs as the graphic symbols in written languages isfound in many parts of the world. An ideograph, as used herein, is agraphic symbol used to represent an object, an idea or a word, withoutexpressing, as in a phonetic system, the specific sounds forming theverbal expression of the object, idea or word. Ideographic languagesinclude Chinese, Japanese and Korean. A graphic symbol, as used herein,includes, but is not limited to, each of the following: a letter of analphabet, a Japanese kana, and an ideograph. For purposes ofillustrating the concepts of the present invention specific referencewill be made herein to a preferred embodiment of the system and methodas it applies to the Chinese language.

In modern Chinese, a repertoire of between 2500 and 3000 ideographs isnecessary to achieve normal business adequacy in reading and writing,while the language itself has approximately 50,000 ideographs that havebeen identified historically, with about 10,000 ideographs in currentuse. The conventional keyboard, with approximately 100 keys, is designedfor languages with phonetic scripts, such languages having a small setof graphic symbols, i.e. letters. If such a keyboard were to be used ina corresponding manner for the direct input of Chinese ideographs, itwould require many thousands of keys since, unlike western phoneticlanguages, Chinese has many thousands of ideographs. Selection of anideograph from such a keyboard would require the operator to search agreat many keys for the desired key, and thus be impracticably slow.

Prior art methods for selecting Chinese ideographs make use of variousideograph classification systems known to Chinese speakers. The operatorfirst specifies a class of ideograph, based on a first characteristiccommon to many ideographs. Ideographs having that common characteristicare displayed and the operator selects from among them, either directly,by selecting an individual ideograph, or indirectly, by specifying asecond common characteristic usually dependent upon the firstcharacteristic, thus further limiting the displayed ideographs to thosehaving both the first and second common characteristics. In some priorart methods, the operator may continue to specify characteristics untilhe has specified a unique ideograph.

One ideograph classification system is called the Pin Yin System. Thisclassification system uses the phonetic structure of the Chineselanguage. In spoken Chinese there are approximately 412 basic phoneticunits, each having a monosyllabic sound, for example, "nee", "how" and"ma". Four intonations can potentially be applied to each phonetic unit,resulting in approximately 1280 distinct sounds. With 10,000 ideographsin current use, each represented by one of approximately 1280 distinctsounds, it is evident that many Chinese ideographs are homophones, i.e.have the same sound. Over 80% of Chinese ideographs have homophones. ThePin Yin System uses this limited number of phonetic units as the basisfor its classification. Ideographs which are homophones are classifiedtogether; the common characteristic of the Pin Yin System is thedistinct sound.

According to the Pin Yin and Zhu Yin coding methods, known in the priorart, the operator specifies a distinct sound using a keyboard labeledwith symbols representing the Latin alphabet (Pin Yin method) or Chinesephonetic units (Zhu Yin method). The first key operation or sequence ofkey operations specifies the phonetic unit. The second key operationspecifies the intonation. In general, less than 15 ideographs have thissound, though in some cases there are many more homophones. These aredisplayed and the user selects from among them. In such cases, theoperator, depending upon the system, may page through matchingideographs or specify another common characteristic to further limit thenumber of ideographs displayed. A common characteristic which may beused at this stage exploits another feature of the Chinese language. Themajority of Chinese words are expressed by a combination of twoideographs, the meaning of the paired ideographs has its own meaningwhich may or may not be related to that of the constituent ideographs.Assuming the operator has specified a first distinct sound matching 40ideographs, he may specify a second distinct sound which alone maymatch, for example, 20 ideographs, but there may be only two ideographpairs having the specified first and second distinct sounds in thatorder. Thus, a second common characteristic may limit matching ideographpairs to a number sufficiently small for the operator to efficientlysearch and select from, or may uniquely specify an ideograph pair.Another common characteristic the operator may specify to limit thenumber of matching ideographs is a meaning or meaning class to which oneor more sequences of one or more ideographs belong.

Yet another feature of the Chinese language which may be exploited tolimit the number of matching sequences of ideographs is the ideographblock. An ideograph block is a sequence of four ideographs whichtogether has its own meaning which may or may not be related to that ofthe constituent ideographs. As above, where the operator specified adistinct sound for the second of two ideographs of an ideograph pair, somay the operator specify a distinct sound for the second, third and/orfourth ideograph of an ideograph block, to limit the number of matchingideograph blocks.

Another conventional ideograph classification system makes use of aclassification of parts of ideographs. Ideographs are built from a setof 214 components, called radicals. Different radicals, perhaps placedwithin different locations within an ideograph, are combined to createan ideograph. According to the Chan Jie coding method, known in theprior art, the operator specifies one or more radicals appearing in theideograph he wishes to enter. He may, for example, use a keyboard havingat least 214 keys, each corresponding to a radical, or may actuate asequence of keys, the sequence corresponding to a radical. Other commoncharacteristics the operator may specify to limit the number of matchingideographs include a phonetic unit, the first brush stroke, and the lastbrush stroke used to draw the ideograph.

Another conventional ideograph classification system makes use of aclassification of parts of ideographs. According to the Four Cornercoding method, known in the prior art, the operator specifies theclassification of the four corners of the ideograph he wishes to enter.Other common characteristics the operator may specify to further limitthe number of matching ideographs include the number of horizontalstrokes used to draw the ideograph, and the classification of a certainpart of the ideograph above the lower right corner.

Yet another conventional ideograph classification system makes use of aclassification ideographs based on the basic strokes from which eachideograph is built. In Chinese, there are a limited number of basicstrokes, each ideograph being composed of between 1 and 33 such strokes.Ideographs may be classified by a small number of basic strokes,preferably according to strict rules regarding the order of strokeentry. In one conventional application of this coding method, theoperator specifies only the first and last basic strokes of the desiredideograph, then selects from a display of all ideographs sharing thisfirst-last basic stroke combination.

Japanese is somewhat more complicated than Chinese. In addition toideographs, the Japanese language uses graphic symbols called kana,which includes hiragana and katakana. In written Japanese, ideographsare frequently combined with kana. Kana may be may specifiedphonetically, for example, to designate the hiragana pronounced "ko" anoperator of a Japanese word processing system may type "k" and then "o"on a Latin alphabetic keyboard or may type a single key associated withthis hiragana. Kana has multiple uses in a Japanese word processingsystem. Kana may represent itself, since kana may stand alone inJapanese text. Alternatively, kana may be used to specify Japaneseideographs, either by specifying the radicals which compose Japaneseideographs or by specifying the pronunciation of Japanese ideographs. Asequence of phonetic units specified by kana may represent that sequenceof kana, a single Japanese ideograph, multiple Japanese ideographs, or acombination of one or more Japanese ideographs and one or more kana. Inaddition, a single Japanese ideograph may have multiple pronunciations,including a Japanese pronunciation and a Chinese pronunciation, and mayhave multiple kana spellings.

Conventional word processing systems for ideographic languages sufferfrom certain deficiencies. First, in systems where the operatorspecifies common characteristics until he has uniquely specified anideograph, the operator must be extensively trained in the particularclassification system. Depending upon the system, the operator may needto know, for example, how may horizontal brush strokes are required todraw a desired ideograph, or each of the 214 radicals and the encodingof each of them on a keyboard having less than 214 keys. Second, insystems where the operator uses both hands on the keyboard to specify acommon characteristic, then selects from among ideographs, ideographpairs or ideograph blocks by operating a function key or by pointing toone of the options with a mouse or other hand operated pointer and thenoperating a switch, the operator lifts one of his hands from thekeyboard, makes the selection and then moves his hand back to thekeyboard to specify another common characteristic. This sequence occursoften and contributes to the slow average rate of word entry(approximately 20 words per minute) for Chinese relative to alphabeticlanguages. Another problem in these systems is that the display ofideographs for selection may obscure part of the image of the previouslyentered ideographs or other information on the display.

Another drawback of many word processing systems for ideographiclanguages relates to the ease of copying a document. Ideally, theoperator concentrates on the document to be copied, only occasionallyscanning text he has input. For those word processing systems thatdisplay ideographs on a display for the operator's selection, theoperator must frequently shift his gaze from the document to the displayand back again. The operator cannot concentrate on both the document andthe display simultaneously.

Ideographs, as used herein, also include the symbols of symbol sets usedfor communication by individuals who have hearing, speech or languageimpairments, for teaching literacy skills to those lacking them,including pre-literate children and individuals with intellectualdisabilities, and for international written communication. These symbolsets include, but are not limited to, each of the following: PictureCommunication Symbols, Rebus, Picsym, Pictogram Ideogram CommunicationSymbols, Yerkish, Blissymbolics and depictions of the signs of a manualsign language. Examples of symbols of the Picture Communication Symbols,Rebus, Picsyms, and Blissymbolics symbol sets are shown in FIG. 11,Pictogram Ideogram Communication Symbols in FIGS. 12(a)-12(d) andYerkish in FIGS. 13(a)-13(j). Picture Communication Symbols, Rebus,Picsyms, Pictogram Ideogram Communication Symbols, Yerkish, andBlissymbolics are each described in Beukelman, David R. & Mirenda, Pat,Augmentative and Alternative Communication. Management of SevereCommunication Disorders in Children and Adults, Paul H. BrookesPublishing Co., 1992, pp. 22-29.

Individuals who have not acquired or who have lost their literacy skillsmay use symbolic symbol sets in learning to read. If the individuallacks fine motor control, for example, due to cerebral palsy, theindividual's disability may inhibit the acquisition of literacy skillsby, for example, inhibiting repetition of an exercise by the individual,by limiting the individual's ability to participate in the classroom, orby making skill assessment by a teacher difficult so that the teachermay incorrectly believe that remediation is necessary or that aparticular skill has been mastered. If the individual also has impairedspeech, literacy acquisition is more difficult still.

Conventional literacy training systems for individuals who are unable touse a standard keyboard or mouse may use switch access, often incombination with scanning. As already described, scanning is anextremely slow process. Moreover, as the number of symbols in the symbolset increases, the time required to select a symbol also increases. Ofthe symbol sets mentioned above, Picture Communication Symbols containsapproximately 1800 symbols, Rebus contains approximately 800 symbols,Picsyms contains approximately 1800 symbols, Pictogram IdeogramCommunication Symbols contains approximately 400 symbols andBlissymbolics contains approximately 1400 symbols. When using a systemwith a static display, the operator may expend considerable time andeffort finding the desired symbol; when using a system with a dynamicdisplay, the operator may expend considerable time effort memorizing andrecalling where a particular symbol is located within a hierarchy ofsymbols. This time and effort generally does not contribute to theacquisition of literacy skills.

One object of the invention is to display a menu of sequences of one ormore ideographs on a display so that a large contiguous area on thedisplay is not obstructed by the menu.

Another object of the invention is to facilitate ideograph entry in wordprocessing systems for the Chinese, Japanese and Korean languages.

Still another object of the invention is to speed selection of sequencesof graphics including one or more ideographs.

Yet another object of the invention is to allow an operator of a wordprocessing system for an ideographic language to select a sequence ofone or more ideographs without lifting either hand from the keyboard.

A further object of the invention is to indicate to an operator theprogress toward selection of a dwell-selectable sequence of one or moreideographic characters.

A still further object of the invention is to synthesize speech for anoperator having impaired speech.

Additional objects, advantages and novel features of the invention willbe set forth in part in the description which follows, and in part willbecome apparent to those skilled in the art upon examination of thefollowing or may be learned by practice of the invention. The objectsand advantages of the invention may be realized and attained by means ofthe instrumentalities and combinations particularly pointed out in theappended claims.

SUMMARY OF THE INVENTION

According to the present invention, one or more of the objects mentionedin the Background Art section describing the Sound Match aspect of theinvention and other objects and advantages are attained by a method ofselecting a sequence of one or more graphic symbols from a plurality ofsequences of one or more graphic symbols, one or more sequences of theplurality of sequences including one or more ideographs in anideographic language. The method includes several steps. The first stepis displaying on a display the plurality of sequences. Each of theplurality of sequences has a common characteristic in the ideographiclanguage and each is associated respectively with one of a plurality ofdisplayed sound indicators. Each of the sound indicators respectivelyindicates one of a plurality of indicated sounds. Each of the indicatedsounds differs from each other indicated sound. Another step is matchinga received sound to any one of the indicated sounds. Yet another step isselecting the sequence associated with the matched indicated sound.

Additionally, one or more of the objects mentioned in the Background Artsection describing the Sound Match aspect of the invention and otherobjects and advantages are attained by a method of selecting a sequenceof one or more graphic symbols in a homophone-rich language from aplurality of sequences of one or more graphic symbols in the language.The method includes several steps. One step is displaying the pluralityof sequences. Each of the plurality of sequences is associatedrespectively with a sound which is not a phonetic representation of theassociated sequence. Each of the plurality of sounds differs from eachother sound in the plurality of sounds. Another step is receiving asound signal. Still another step is matching the received sound signalto any one of the plurality of sounds. Yet another step is selecting thesequence associated with the matched sound.

Additionally, one or more of the objects mentioned in the Background Artsection describing the Sound Match aspect of the invention and otherobjects and advantages are attained by a method of editing a firstsequence of two or more words in an ideographic language. The methodincludes several steps. One step is displaying on a display the firstsequence. The first sequence includes a plurality of second sequences ofone or more graphic symbols in the language. Each of the secondsequences is associated respectively with a sound indicator. Each of thesound indicators respectively indicates a sound. Each of the indicatedsounds differs from each of the other indicated sounds; and (b) thesound indicators. Another step is receiving a sound. Still another stepis matching the received sound to any one of the indicated sounds. Afurther step is replacing the second sequence associated with thematched indicated sound with a third sequence of one or more graphicsymbols in the language, the third sequence including an ideograph inthe language.

Additionally, one or more of the objects mentioned in the Background Artsection describing the Sound Match aspect of the invention and otherobjects and advantages are attained by an apparatus, for use with ageneral purpose computer system including a display, for selecting asequence of one or more graphic symbols from a plurality of sequences ofone or more graphic symbols. One or more sequences of the plurality ofsequences includes an ideograph in an ideographic language. Theapparatus includes a computer readable medium, and a program, stored onthe medium and executable by the general purpose computer system. Theprogram is operative to display the plurality of sequences on thedisplay. Each of the sequences is associated respectively with adisplayed sound indicator. Each of the sound indicators respectivelyindicates a sound. Each of the plurality of indicated sounds differsfrom each of the other indicated sounds. A particular one of the soundindicators is not a phonetic representation of the sequence associatedwith the particular sound indicator. The program is further operative tomatch a received sound to the sound indicated by the particular soundindicator, and to select the sequence associated with the particularsound indicator.

Additionally, one or more of the objects mentioned in the Background Artsection describing the Sound Match aspect of the invention and otherobjects and advantages are attained by an apparatus for selecting asequence of one or more graphic symbols from a plurality of sequences ofone or more graphic symbols, one or more sequences of the plurality ofsequences including one or more ideographs in an ideographic language.The apparatus includes a display on which may be displayed the pluralityof sequences. Each of the plurality of sequences has a commoncharacteristic in the ideographic language and each is associatedrespectively with a displayed sound indicator. Each of the soundindicators respectively indicates a sound. Each of the plurality ofindicated sounds differs from each of the other indicated sounds. Aparticular one of the sound indicators is not a phonetic representationof the sequence associated with the particular sound indicator. Theapparatus further includes a selection device for matching a receivedsound to the sound indicated by the particular sound indicator and forselecting the sequence associated with the particular sound indicator.

Additionally, one or more of the objects mentioned in the Background Artsection describing the Sound Match aspect of the invention and otherobjects and advantages are attained by an apparatus for selecting asequence of one or more graphic symbols from a plurality of sequences ofone or more graphic symbols, one or more sequences of the plurality ofsequences including one or more ideographs in an ideographic language.The apparatus includes means for displaying the plurality of sequences.Each of the plurality of sequences has a common characteristic in theideographic language and each associated is respectively with adisplayed sound indicator. Each of the sound indicators respectivelyindicates a sound. Each of the plurality of indicated sounds differsfrom each of the other indicated sounds. A particular one of the soundindicators is not a phonetic representation of the sequence associatedwith the particular sound indicator. The apparatus further includesmeans for: (1) matching a sound to the sound indicated by the particularsound indicator; and (2) selecting the sequence associated with theparticular sound indicator.

Additionally, one or more of the objects mentioned in the Background Artsection describing the Sound Match aspect of the invention and otherobjects and advantages are attained by a device controller. The devicecontroller includes a display on which may be displayed a plurality ofsequences of one or more graphic symbols, one or more sequences of theplurality of sequences including one or more ideographs in anideographic language. Each of the plurality of sequences has a commoncharacteristic in the ideographic language and each is associatedrespectively with a displayed sound indicator. Each of the soundindicators respectively indicates a sound. Each of the plurality ofindicated sounds differs from each of the other indicated sounds. Aparticular one of the sound indicators is not a phonetic representationof the sequence associated with the particular sound indicator. Thedevice controller further includes a signal generating device, coupledto a controlled device, for: (1) matching a sound to the sound indicatedby the particular sound indicator, the sequence associated with theparticular sound indicator representing a function of the controlleddevice; and (2) generating a device control signal corresponding to thefunction of the controlled device.

Still other objects and advantages of the present invention will becomereadily apparent to those skilled in this art from the followingdetailed descriptions, wherein I have shown and described the preferredembodiment of each aspect of the invention, simply by way ofillustration of the best mode contemplated by me of carrying out eachaspect of my invention. As will be realized, each aspect of theinvention is capable of other and different embodiments, and its severaldetails are capable of modifications in various obvious respects, allwithout departing from the invention. Accordingly, the drawings anddescriptions are to be regarded as illustrative in nature, and not asrestrictive.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an illustration of a display showing a conventional on-screenkeyboard.

FIG. 2 is an illustration of a display showing a conventional on-screenkeyboard and output from a word processing application program.

FIGS. 3, 4 and 5 are each illustrations of the display of each step ofletter selection using a conventional quaternary on-screen keyboard.

FIG. 6 is an illustration of a display showing a conventional quaternaryon-screen keyboard and output from two application programs.

FIG. 7 is an illustration of a conventional device implementing joystickpatterns.

FIG. 8 is an illustration of a display of a conventional eye gazesystem.

FIGS. 9 and 10 are each illustrations of a display showing aconventional menu.

FIG. 11 depicts examples of symbols from the Picture CommunicationSymbols, Rebus, PicSym and Blissymbols symbol sets.

FIGS. 12(a)-FIG. 12(d) depict examples of symbols from the PictogramIdeogram Communication Symbols symbol set.

FIGS. 13(a)-FIG. 13(j) depict examples of symbols from the Yerkishsymbol set.

FIG. 14 is a block diagram of a computer which may be utilized inaccordance with the present invention.

FIG. 15 is a block diagram of a speech synthesis system which may beutilized in accordance with the present invention.

FIG. 16 is an illustration of software components of an apparatus inaccordance with an embodiment of the Perimeter Menu aspect of theinvention.

FIGS. 17 and 18 are each illustrations of a display and structures inaccordance with an embodiment of the Perimeter Menu aspect of theinvention.

FIG. 19 is an illustration of a display and structures in accordancewith another embodiment of the Perimeter Menu aspect of the invention.

FIG. 20 is an illustration of a display and structures in accordancewith still another embodiment of the Perimeter Menu aspect of theinvention.

FIG. 21 is an illustration of a display and structures in accordancewith another embodiment of the Perimeter Menu aspect of the invention.

FIG. 22 is an illustration of a display and structures in accordancewith yet another embodiment of the Perimeter Menu aspect of theinvention.

FIG. 23 is an illustration of an apparatus in accordance with a furtherembodiment of the Perimeter Menu aspect of the invention.

FIGS. 24 and 25 are each illustrations of a display and structures inaccordance with another embodiment of the Perimeter Menu aspect of theinvention.

FIGS. 26 and 27 illustrate an apparatus in accordance with still anotherembodiment of the Perimeter Menu aspect of the invention. FIG. 26depicts a front view of the apparatus. FIG. 27 depicts a cut away viewfrom the top of the apparatus.

FIG. 28 is a top view of a headrest in accordance with an embodiment ofthe Perimeter Menu aspect invention.

FIGS. 29 and 30 illustrate the state table aPocketFsm in accordance withthe preferred embodiment of the Perimeter Menu aspect of the invention.

FIG. 31 is an illustration of a display and structures in accordancewith the preferred embodiment of the Confinement aspect of theinvention.

FIG. 32 is an illustration of a display and structures in accordancewith another embodiment of the Confinement aspect of the invention.

FIG. 33 is an illustration of a display and structures in accordancewith another embodiment of the Confinement aspect of the invention.

FIG. 34 is an illustration of an apparatus in accordance with the Dwellaspect of the invention and with the Path Directness aspect of theinvention.

FIG. 35 is an illustration of another apparatus in accordance with theDwell aspect of the invention.

FIG. 36 is an illustration of still another apparatus in accordance withthe Dwell aspect of the invention.

FIG. 37 is an illustration of yet another apparatus in accordance withthe Dwell aspect of the invention.

FIG. 38 is an illustration of another apparatus in accordance with theDwell aspect of the invention.

FIG. 39 is an illustration of a display and structures in accordancewith the preferred embodiment of the Path Directness aspect of theinvention.

FIG. 40 is an illustration of a display and structures in accordancewith an embodiment of the Path Directness aspect of the invention.

FIG. 41 is an illustration of a display and structures in accordancewith another embodiment of the Path Directness aspect of the invention.

FIG. 42 is an illustration of a display and structures in accordancewith another embodiment of the Path Directness aspect of the invention.

FIG. 43 is an illustration of a display and structures in accordancewith another embodiment of the Path Directness aspect of the invention.

FIGS. 44, 45 and 46 are each illustrations of a display and structuresin accordance with the preferred embodiment of the Intersection aspectof the invention.

FIGS. 47 and 48 are each illustrations of a display and structures inaccordance with another embodiment of the Intersection aspect of theinvention.

FIGS. 49, 50 and 51 are each illustrations of a display and structuresin accordance with the preferred embodiment of the Alignment aspect ofthe invention.

FIG. 52 is an illustration of a display and structures in accordancewith the preferred embodiment of the Location Indication and the LengthOrder aspects of the invention.

FIG. 53 is an illustration of a display and structures in accordancewith an embodiment of the Location Indication aspect of the invention.

FIGS. 54 and 55 are each illustrations of a display and structures inaccordance with another embodiment of the Location Indication aspect ofthe invention.

FIG. 56 is an illustration of a display and structures in accordancewith a still further embodiment of the Location Indication aspect of theinvention.

FIG. 57 is an illustration of a display and structures in accordancewith a further embodiment of the Location Indication aspect of theinvention.

FIGS. 58 and 59 are illustrations of a display and structures inaccordance with the preferred embodiment of the Sound Match aspect ofthe invention.

FIG. 60 is an illustration of a display and structures in accordancewith another embodiment of the Sound Match aspect of the invention.

FIG. 61 is an illustration of a display and structures in accordancewith another embodiment of the Sound Match aspect of the invention.

FIG. 62 is a block diagram of a speech recognition system which may beutilized in accordance with the Sound Match aspect of the invention.

FIG. 63 is an illustration of software components of an apparatus inaccordance with an embodiment of the Sound Match aspect of theinvention.

FIG. 64 is an illustration of a display and structures in accordancewith the preferred embodiment of the Ideographic Language aspect of theinvention.

FIG. 65 is an illustration of a display and structures in accordancewith another embodiment of the Ideographic Language aspect of theinvention.

FIG. 66 is an illustration of a display and structures in accordancewith still another embodiment of the Ideographic Language aspect of theinvention.

FIG. 67 is an illustration of a display and structures in accordancewith yet another embodiment of the Ideographic Language aspect of theinvention.

BEST MODE FOR CARRYING OUT THE INVENTION

The hardware and software operating environment of the preferredembodiment of all aspects of the invention will now be described withreference to a particular embodiment of the invention, hereinafter"prototype". The prototype of the invention illustrates the best mode ofpracticing each aspect of the invention known by me except where apreferred mode is described.

FIGS. 15 depicts a block diagram of the hardware components of theprototype (2214), including a conventional general purpose computersystem (2218), an optional pointer (2202), an optional printer (2220)and a speech synthesizer (2206). The general purpose computer system(2218) includes a conventional computer system (2116), a storage unit(2208), a keyboard (2210), and a diskette drive (2216). FIG. 14 depictsa block diagram of the conventional computer system (2116), including aprocessing unit (2102) and a display (2112). The processing unit (2102)includes a processor (2104), a memory (2106) and control circuitry(2108). The prototype employs the Toshiba T6400DXC general purposecomputer system manufactured by Toshiba Corporation, Kawasaki, Japan.However, the T6400DXC is preferably substituted with the IBM ThinkPad755C computer system, part number 9545F0C, manufactured by IBMCorporation, Armonk, N.Y., USA., because the former requires a 110VACpower source while the latter is powered by an integral battery. Anintegral battery allows an NMD operator to use the system when a 100VACsource is not available or when attaching to a 110VAC power source isinconvenient. The prototype further includes a head mounted pointercommunicating via an infrared link with the computer system so thatthere are no cables tethering the operator to the computer system. Anycable between the operator and the computer system would have to beconnected, probably by an attendant since the operator may lack the finemotor skills required to make such a connection. Preferably computeraccess can be accomplished independently by the operator. The choice ofa pointing device is primarily dictated by the particular capabilitiesof the operator. Usually the best pointing device for a particularoperator is the one drawing on that operator's best motor control. Forexample, if an operator's foot control is superior to his head control,a pointing device using his foot is preferably to a head pointer.

The prototype employs the Remote Headmaster® manufactured by the PrentkeRomich Company, Wooster, Ohio, USA. However, the combination of theHeadMaster® Plus, part number HM-1P, HeadMaster® Plus Remote Adapter,part number HM-RA, and HeadMaster® Plus Laptop Adapter, part numberHM-LA, all available from the Prentke Romich Company, is preferablebecause the headset is more comfortable and the HeadMaster® Plusultrasonic transmitter mounts more easily on a laptop computer systemthan the Remote Headmaster ultrasonic transmitter.

The preferred embodiment further includes a battery powered printer, theMobileWriter®, part number 730879, manufactured by Mannessmann TallyCorporation, Kent, Wash., USA and a speech synthesizer, the MultivoiceSpeech Synthesizer, part number MV2-SS, manufactured by The Institute onApplied Technology, Children's Hospital, Boston, Mass., USA.

The pointer (2202) is a device which provides data concerning therelative or absolute position of the operator or any body member of theoperator. The display (2112) and pointer (2202) together provide for theinteractive nature of the general purpose computer system (2218) inthat, in accord with the various aspects of the invention, theinterpretation that the processor (2104) gives to a certain pointeraction made by the operator depends, in the majority of situations, uponwhat is being displayed to the operator at that time.

The prototype (2214) shown in FIG. 15 further includes a keyboard(2210), which functions to provide input from an able-bodied operator tothe general purpose computer system (2218). The keyboard (2210) isuseful for configuration, diagnostic and backup purposes, functionswhich are performed relatively infrequently and usually require anable-bodied person for ancillary activities, for example, loading backupmedia into the general purpose computer system. The prototype (2214)also optionally includes a printer (2220) which functions to providehard copy output of data developed or stored in the general purposecomputer system, and a speech synthesizer (2206), which functions toprovide speech output for utterances and words composed using orretrieved from the general purpose computer system (2218).

The couplings between the devices depicted in FIG. 15 may be made by anymeans which permits the orderly and timely exchange of data across theinterface. In the preferred embodiment, the interfaces between thepointer (2202) and the general purpose computer system (2218) andbetween the general purpose computer system (2218) and the speechsynthesizer (2206) conform to the Electronic Industries AssociationRS-232 interface specification. The interface between the generalpurpose computer system (2218) and the printer (2220) conform to theCentronix 50 pin parallel interface specification.

The software component of the prototype are stored in memory (2106) andexecuted on the processing unit (2102). The software component of theprototype, depicted in FIG. 16, include a software driver (1202), anoperating system (1204), an optional database program (1210), and theprototype access program code and data, hereinafter collectivelyreferred to as the "access program" (1206). In the preferred embodiment,one or more application programs (1208) may also execute on theprocessing unit (2102) and accept control and data from the accessprogram (1206) via the operating system (1204). The software driver(1202) of the prototype is the Logitech Mouse Driver included withWindows® version 3.1. The operating system (1204) of the prototype isWindows® version 3.1 in combination with MS-DOS® version 6.2.Hereinafter, the operating system is referred to simply as "Windows®",available from Microsoft Corporation, Redmond, Wash., USA.

The optional database program (1210) is described in the detaileddescription of the Length Order aspect of the invention. The prototypeaccess program (1206) is described in detail below.

As stated earlier, the software components of the prototype are storedin memory (2106). Depending on the capacity of memory (2106) and thesize of the application programs, portions of these programs may betransferred as needed between memory (2106) and the storage unit (2206)or between memory (2106) and a diskette in the diskette drive (2216)depicted in FIG. 15. The basic function of the storage unit (2206) andthe diskette drive (2216) is to store programs and data that areemployed by the general purpose computer system (2218) and which mayreadily be transferred to the memory (2106) when needed.

It is to be understood that components others than those used in theprototype may be utilized in accordance with the invention. It is onlynecessary that the substitute component or components have the capacityto carry out the functions described. For example, the processing unitof the general purpose computer system may be substituted with amicroprocessor coupled to custom electronics for performing thefunctions of the various aspects of the invention, or the color displayof the prototype may be substituted with a monochrome display.

A. Perimeter Menu

The preferred embodiment of the Perimeter Menu aspect of the inventionwill now be described in detail from a functional perspective using anexample. This description refers to selectable regions which include oneor more subregions. A selectable region is a region, delimited withrespect to a display or a surface, and associated with a menu optionwhich may be selected, usually by a selection event. A subregion is aselectable region that is included within another selectable region.Thus a subregion is, by itself, a selectable region. Assuming that acertain selectable region includes subregions A and B, dwell time onsubregions A and B may be combined, for example, by summing, so thatdwelling on either subregion A or B or a combination of both for theselection threshold period selects the menu option associated with theselectable region. Reference will now be made to FIGS. 17 and 18 whichdepict an example of the preferred embodiment of the Perimeter Menuaspect of the invention. FIG. 17 shows the display (2112) of a generalpurpose computer system (2218 in FIG. 15) and eight selectable regions.Each of the eight selectable regions consists of the union of a visiblesubregion on the display (2112) and an invisible subregion locatedoutside the display (2112) and adjacent the visible subregion. Forexample, the selectable region at 11 o'clock in FIG. 17 labeled withmenu option "vort<space>x" consists of invisible subregion (0104) andvisible subregion (0106), and within this description of the PerimeterMenu aspect of the invention is referred to as selectable region(0104/0106). The other selectable regions shown in FIG. 17, proceedingcounter clockwise from selectable region (0104/0106) are (0108/0110),(0112/0114), (0116/0118), (0120/0122), (0124/0126), (0128/0130) and(0132/0134). Each subregion may be sized to suit the operator'spreferences and abilities. Each selectable region is associatedrespectively with a menu option. In FIG. 17, selectable region(0104/0106) is associated with menu option vort<space>x, selectableregion (0108/0110) with menu option "sumac", selectable region(0112/0114) with menu option "wizen'", selectable region (0116/0118)with the menu option undo indicated by an icon on visible subregion(0118) representing an undo function, selectable region (0120/0122) withmenu option "words", selectable region (0124/0126) with menu option"talk", selectable region (0128/0130) with menu option "ldhbfk" andselectable region (0132/0134) with menu option "ypgqj,". Together, theeight visible subregions circumscribe region (0150) on the display.

Selectable regions may be delimited by data indicative of one or moreboundaries of the selectable region. Equivalently, the delimit means maybe detectors operative to determine when the location indicated by themovement related signal has crossed one of those boundaries orintersects a selectable region. A partially delimited region orsubregion is one which is unbounded on at least one side.

Resuming, now, with the example, to select a menu option associated witha selectable region the operator moves a pointer (2202 in FIG. 15)coupled to the general purpose computer system (2218 in FIG. 15) toindicate a location on the selectable region, including eithersubregion, associated with the desired menu option and maintains theindicated location on the selectable region for the selection thresholdperiod. The period of time required for selection may vary responsive tothe proximity of the indicated location to the location of a cursor onthe display or to the proximity of the indicated location to a pointwithin the intersected selectable region. Dwell time may be continuous,discontinuous or dynamic (described below) for either or both subregionsof the selectable region.

Selection in the above example is in response to a dwell event. A dwellevent includes, but is not limited to, each of the following: (a) thedurations of one or more periods of intersection of locations indicatedby a movement related signal, a body member or a cursor (including anypart of the cursor) and a selectable region equalling or exceeding apredetermined period; (b) a first quantity responsive to the durationsof the periods referred to in (a) equalling or exceeding a predeterminedquantity; (c) dwell event (a) or (b) followed by a location indicated bythe movement related signal, the body member or the cursor no longerintersecting the intersected selectable region; and (d) dwell event (a)or (b) wherein the period of intersection required for selection of aselectable region increases in response to a non-intersection or aperiod of non-intersection of locations indicated by the movementrelated signal, the body member or the cursor and the selectable region("dynamic dwell event"). The use of non-intersection or a period ofnon-intersection in determining the duration of a period of intersectionrequired for selection is called dynamic dwell. Associated with eachtype of dwell event is an intersected selectable region. This is theselectable region intersected by the location indicated by the movementrelated signal, body member or cursor which triggers the dwell event bycausing the period or the first quantity to equal or exceed thepredetermined period or the predetermined quantity, respectively.

Selection may also be in response to a selection event. A selectionevent includes, but is not limited to: (a) a dwell event; (b) a switchoperation at or near the time of an intersection of a location indicatedby a movement related signal, a body member or a cursor and a selectableregion; (c) an intersection of a location indicated by a movementrelated signal, a body member or a cursor and a selectable region; and(d) selection event (c) followed by a location indicated by the movementrelated signal, the body member or the cursor no longer intersecting theselectable region it previously intersected. Associated with each typeof selection event is an intersected selectable region. This is theselectable region intersected by the location indicated by the movementrelated signal, body member or cursor. The fact that a selection eventhas occurred may be indicated to the operator, for example, visually bychanging the cursor appearance or location, by changing location, size,shape, hue, brightness, contrast, tone, dithering, pattern, hatching,font or fill of an object on the surface, or by displaying a graphic ora point distinguishable from its immediate surroundings on a surface orremoving a graphic or point distinguishable from its immediatesurroundings from a surface; auditively by generating a sound orchanging the pitch or volume of an extant sound; tactilely by changingthe surface or temperature of a contact area or the pressure exerted bya contact area; or by other means. In the prototype, followingselection, the hue of the visible subregion of the selected selectableregion is changed from green to magenta.

As used herein, a cursor includes a temporary marking on a display whichemphasizes to an operator, in an optical manner, a momentarily importantlocation or object. As used herein, body member means any part of thebody including, but not limited to, each of the following: the shoulder,arm, elbow, wrist, hand, finger, thumb, leg, knee, ankle, foot, toe,hip, trunk, neck, tongue, lip, eye and head. The received movementrelated signal includes, but is not limited to, a signal indicative ofmovement or from which movement can be derived, such as a plurality ofrelative or absolute positions or a difference between two relative orabsolute positions. Movement related signal receiving means includes,but is not limited to, each of the following: (a) pointer interfacecircuitry found in a general purpose computer system; (b) one or moredetectors operative to detect movement of a pointer; and (c) one or moredetectors operative to detect movement of a body member of an operator.In the prototype, the movement related signal receiving includeselectronic circuitry in the general purpose computer system (2218)operative to receive the movement related signal generated in part bythe movement of the pointer (2202).

In the prototype subregions are displayed on the display (2112).However, other means for displaying may be substituted for the meansused in the prototype, for example, a projector for projecting an image,a surface having a static display thereon, or other suitable means.

Resuming, now, with the example of FIG. 17, and assuming that theoperator has selected menu option "vort<space>x", the display is changedto that shown in FIG. 18. In FIG. 18, each of six selectable regions isnow associated with a submenu option of the selected menu option"vort<space>x". Selectable region (0104/0106) is now associated withsubmenu option "<space>", selectable region (0108/0110) with submenuoption "o", selectable region (0112/0114) with submenu option "t",selectable region (0124/0126) with submenu option "x", selectable region(0128/0130) with submenu option "v", and selectable region (0132/0134)with submenu option "r". Selectable regions (0116/0118) and (0120/0122)remains associated with the same menu options with which each wasassociated in FIG. 17. The operator may now select one of these submenuoptions.

Assuming that the selected submenu option is one of v, o, r, t, x andspace, the selected character, or a corresponding computer encoding ofthat character, may be input to an apparatus coupled to the generalpurpose computer system (2218 in FIG. 15), or input to an applicationprogram (1208) executing on the general purpose computer system (2218)coupled to the display (2112). Inputting, as used herein, includes, butis not limited to, generating or passing signals representative of theselected menu option along a path toward the destination apparatus orprogram. Preferably, the computer program displays at least some of itsoutput in the circumscribed region (0150).

Given a display having eight selectable regions, an operator may, with asingle selection indicate one of eight menu options, with two selectionsindicate one of up to 64 different menu options, with three selectionsindicate one of up to 256 menu options, etc. Each of these menu optionsmay represent a sequence of one or more characters, a sequence of one ormore data or control inputs to an application program (1208), or acontrol function for one or more devices or speech synthesizers coupledto the general purpose computer system (2218). As used herein, acharacter includes a space, a control character as defined by theAmerican National Standards Institute (ANSI) or the American StandardCode for Information Exchange (ASCII), and a letter from one of theAfrikaans, Albanian, Amharic, Arabic, Armenian, Assamese, Assyrian,Avar, Azerbaijani, Balinese, Bamara, Bantu, Bashkir, Basque, Bengali,Birhari, Bulgarian, Buluba-Lulua, Burmese, Buryat, Byelorussian,Caddoan, Catalan, Chechen, Chikaranga, Chippewa, Choctaw, Church Slavik,Chuvash, Coptic, Cree, Croatian, Cyrillic, Czech, Dakota, Danish, Dari,Devanagari, Dutch, Dzongkha, English, Eskimo, Esperanto, Estonian, Ewe,Farsi, Fijian, Filipino, Finnish, Flemish, French, Fulani, Gaelic,Galician, Georgian, German, Greek, Gujarati, Gurmakhi, Harari, Hausa,Hawaiian, Hebrew, Hindi, Hiragana, Ibo, Icelandic, Indonesian, Irish,Irogquoian, Italian, Kabardian, Kalmyk, Kannada, Kanuri, Kashmiri,Katakana, Kazakh, Khasi, Khmer, Kirghiz, Kishmiri, Komi, Kongo, Kurdish,Lao, Latin, Latvian, Lithuanian, Lu-Guanda, Macedonian, Magahi,Maithili, Makua, Malagasy, Malay, Malayalam, Maltese, Mandingo,Manipuri, Marathi, Masai, Mizo, Moldavian, Mongolian, Munda, Naga,Navaho, Nyanja, Nepalese, Norwegian, Oriya, Oromo, Ossetian, Pashto,Polish, Portugese, Punjabi, Rajasthani, Rhaeto-Romanic, Rumanian,Russian, Samoan, Sangs, Serbian, Serbo-Croatian, Sinhalese, Sinhi,Sioux, Slovak, Slovenia, Spanish, Sundanese, Swahili, Swedish, Syriac,Tadzhik, Tagalog, Tajik, Tamil, Tatar, Telugu, Thai, Tibetan, Turkish,Turkmen, Udmurt, Uighur, Ukranian, Umbundu, Urdu, Uzbek, Vietnamese,Visayan, Welsh, Yakut, Yoruba and phonetic alphabets. As used herein,each of a character, ideograph, control input and control functionincludes a computer encoding of the same. As used herein, a deviceincludes, but is not limited to, each of a wheelchair, a householdappliance, an appliance for use in an office, a workstation, a robot,and a computer peripheral. Thus, by selecting from a menu, the operatormay, for example, increase the volume of an external speech synthesizer,or turn a wheelchair to the left.

The selectable regions organized as described above help an NMD operatormake the menu selection he intends. Referring to FIG. 17, suppose, forexample, an NMD operator intends to move a pointer (2202) that isindicating point (0154) to indicate point (0156), a location insubregion (0130), but who is unable to quickly stop motion, so that thelocation indicated by the pointer (2202) moves from point (0154) pastpoint (0156) to point (0158). Because point (0158) lies within the sameselectable region (0128/0130) as the overshot subregion (0130), dwellingat point (0158) operates to select the intended selectable region(0128/0130). Invisible subregions, in accordance with the Perimeter Menuaspect of the invention, may extend outward from the edge of the display(2112) to infinity. In such an embodiment, dwelling at point (0162)would operate to select selectable region (0128/0130). Preferably,invisible subregions extend a finite distance from the edge of thedisplay (2112). In such an embodiment, dwelling at point (0162) wouldnot operate to select selectable region (0128/0130). The sizes of theinvisible subregions shown in FIGS. 17 and 18 are illustrative only.Preferably the size of each invisible subregion is large enough toencompass overshoot but small enough to avoid unintentional selectionswhen the NMD operator turns to see someone or something.

The prototype utilizes the area outside the display to facilitate menuselection by a disabled operator. If an operator has impaired ability tomaintain a steady position, he can point to a relatively large invisiblesubregion outside the display which is more forgiving of the operator'sinvoluntary motion than the relatively small selectable regions on thedisplay in conventional on-screen keyboards. Thus, the effective area ofa selectable region is expanded beyond the region's visible subregionshown on a display. If an operator has impaired ability to stop motionhe may, starting from the center of the screen, point to any selectableregion. His impaired ability to stop will not impair his ability toselect his intended target, assuming his directional control isrelatively unimpaired, since, in the prototype, each selectable regionis unbounded on its side furthest from the center of the display.Although the selectable regions of the prototype are large, only thevisible subregion of each selectable region uses space on the display.Thus a large rectangular region remains available on the display for theoutput of an application program. The Perimeter Menu aspect of theinvention is preferably implemented on a general purpose computersystem. If the general purpose computer system is coupled to a speechsynthesizer and the menu hierarchy allows the selection of lettersand/or words, an operator having impaired speech may speak using thespeech synthesizer. If a word processing or data entry applicationprogram is run on the general purpose computer system, the operator mayenter words or data, respectively, for input to the application program.If the general purpose computer system is coupled to a devices capableof executing commands and the menu hierarchy allows the selection ofcommands, a disabled operator may select and issue commands to controlthese devices.

Since the cognitive demand for scanning is greater than that for directselection, the prototype places less cognitive demand on the operatorthan a scanning system for selecting options from a menu.

FIG. 19 illustrates a display in accordance with an alternativeembodiment of the Perimeter Menu aspect of the invention having eightselectable regions circumscribing a central region (6950) on thedisplay.

FIG. 20 illustrates a display in accordance with an embodiment of thePerimeter Menu aspect of the invention having twenty selectable regionscircumscribing a central region (0806) on the display.

FIG. 21 illustrates a display in accordance with an embodiment of thePerimeter Menu aspect of the invention having four selectable regions(0508), (0506), (0504) and (0502) circumscribing a central region (0510)on the display.

FIG. 22 illustrates a display in accordance with an embodiment of thePerimeter Menu aspect of the invention having four selectable regions(4008), (4006), (4004) and (4002) with no space between them other thana circumscribed region (4010) on the display.

FIG. 23 illustrates an apparatus in accordance with of anotherembodiment of the Perimeter Menu aspect of the invention. In FIG. 23,region (3510) is located on interior display (3514) which iscircumscribed by peripheral display (3512). Selectable regions (3508),(3535), (3504) and (3502) are located on the peripheral display (3512).

FIGS. 24 and 25 are each illustrations of a display and structures inaccordance with another embodiment of the Perimeter Menu aspect of theinvention. FIG. 24 depicts ten selectable regions (6502), (6504),(6506), (6508), (6510), (6512), (6514), (6514), (6516), (6518), and(6520). Each selectable region is located on the display (2112) adjacentthe edge of the display and associated respectively with a menu option.In FIG. 24, the menu options are shown on their associated selectableregion. Together the ten selectable region circumscribe region (6550) onthe display. In response to only an intersection of a location indicatedby a movement related signal and selectable region (6506), the displaychanges to that shown in FIG. 25, on which are located ten selectableregions each located on the display (2112) adjacent the edge of thedisplay, nine of the ten selectable regions associated respectively witha submenu option. Selectable region (6608) is not associated with asubmenu option. Any submenu option may be selected by a selection event.

FIGS. 26 and 27 illustrate an apparatus in accordance with still anotherembodiment of the Perimeter Menu aspect of the invention. FIG. 26depicts a front view of the apparatus; FIG. 27 a cut away view from thetop of the apparatus. FIG. 26 depicts detector area (0348 in FIG. 27)on, in or below which are located a plurality of selectable regions(0304), (0306), (0308), (0310), (0312), (0314), (0316), (0318), (0320),(0322), (0324), (0326), (0328), (0330), (0332), (0334), (0336), (0338),(0340) and (0342). Adjacent the detector area (0348 in FIG. 27) is aberm (0350 in both FIG. 26 and 27) for confining a body member of theoperator or a pointer controlled by the operator to the detector area(0348 in FIG. 27).

Still another apparatus in accordance with the Perimeter Menu aspect ofthe invention is illustrated in FIG. 28 which depicts a headrest for anoperator using his head to indicate a location on a display. FIG. 28shows an irregularity (0703) on the surface (0701) of the headrest. Theirregularity is in physical contact with the operator and tactilelyindicates to him the position of his head. The tactile indication meansmay be concave, convex or both or may differ from the surface intemperature. For individuals having impaired ability to sense theposition of a body member, e.g. the operator's head, the tactile inputthus provided to the operator improves the operator's ability to sensethe position of his head.

The prototype of the invention will now be described in detail and wherethe preferred mode of practicing the invention differs from theprototype, the preferred mode is described. The description is brokeninto several parts:

1. A brief overview of how a state table works.

2. A description of the operation of the events, state table and stateprocessing used in the prototype.

3. A general description of each event and one example of the use ofthat event.

4. An example of state machine processing in the operation of theprototype.

The prototype implements the Perimeter Menu aspect of the invention as astate table. A state table is a tool for processing sequential inputsand is most easily understood by analogy. Imagine yourself in a roomhaving a ticket window and three exits, each regulated by a turnstile.You collect a ticket at the ticket window which, when inserted into theappropriate turnstile, allows passage to a connecting room. Theturnstile keeps the ticket. Any given ticket operates only one turnstilein a room, though different tickets may operate the same turnstile. Youbegin in a certain room, collect a ticket, insert it into theappropriate turnstile and pass to a connecting room, where you performcertain tasks associated with the new room. Then you collect anotherticket from the ticket window in that room, insert the ticket into theappropriate turnstile in that room, pass to a connecting room, performcertain tasks associated with the new room, and so on.

The state table used in the prototype is depicted in FIGS. 29 and 30.Moving from the analogy above, the rooms are states represented by therows of the state table; the tickets are events represented by thecolumns of the state table. Each entry in the state table represents apassage from one state to another. The tasks performed upon entry into aroom correspond to the processing performed by the processor (2104) onentry to a new state ("state processing"). For example, state processingmay cause a slight lightening of the color of a selectable region.Reentry into that state five times may successively lighten a selectableregion five times.

In the prototype, each selectable region is directly controlled by oneassociated state machine. Each state machine directly controls only itsassociated selectable region. Each state machine includes data uniquelyassociated with its associated selectable region, the shared state tableshown in FIGS. 29 and 30, and the shared code for state processing,described below. This embodiment means that the state machine associatedwith selectable region (1614) may be in state ST₋₋ SELECTED while thestate machine associated with selectable region (1602) is in state ST₋₋INITIAL. In the prototype, these separate states are reflected only inseparate values for data uniquely associated with each state machine.Each state machine has a unique index.

The state table used in the prototype defines 18 states, composed ofstates zero through seventeen shown in FIGS. 29 and 30. State 1 (ST₋₋ERROR₋₋ STATE in FIG. 29) is not used. Preferably, it is omitted. In theprototype, the differences between the several state machines, forexample, the state of a particular state machine at any given time, areconfined to data structures associated with that state machine. Thestate machines share the same code and the same state table. In otherwords, returning to the ticket and turnstile analogy, there are multipletravelers each with his own ticket (event) and his own baggage (data)moving from room to room in the same labyrinth. Each traveler's actionsin each room usually affect only his own baggage.

In the prototype, there are two types of sequential inputs processed bythe state table: external events and internal events. External eventsare generated outside the state table, for example, by the operatormoving the pointer (2202) or by a timer expiring. Pointer movement maygenerate an event indicating that the operator has moved the cursoracross the selectable region boundary from without the selectable regionto within it. This event causes a transition from one state to another("drives" a state machine to a new state). For example, assuming a statemachine is in state ST₋₋ CREST₋₋ TIDE, row 6 in the state table shown inFIG. 29, when an event EV₋₋ CROSS₋₋ OUT, column 4 of the state table,occurs. At the intersection of row 6 and column 4 is a 7. Thisrepresents the new state, row 7 of the state table, state ST₋₋ SELECTED.Thus event EV₋₋ CROSS₋₋ OUT drives the state machine from state ST₋₋CREST₋₋ TIDE to state ST₋₋ SELECTED. On entry to the new state, thecomputer performs the state processing associated with the new state.

Internal events are generated during state processing to handlecircumstances where a first state transition is made due to an externalevent and the processing associated with the new state determines that asecond state transition is necessary. The first, external, event hasalready been used so a second, internal, event is generated by thestate. For example, if the operator has a prolonged muscle spasm, commonamong individuals with CP, or loses his grip on a hand held pointingdevice, the cursor may sit without moving on a selectable region for aconsiderable period of time. It is desirable to detect this condition,move the cursor to the center of the screen so the operator can easilyfind the cursor, and reset all selectable regions to their initialcolor. Detection is accomplished with a timer. On timer expiration, astate machine transitions to state ST₋₋ IDLE which centers the cursor.From there it is desirable to transition to state ST₋₋ RESET which,among other processing, initializes selectable region color. Thetransition from state ST₋₋ IDLE to state ST₋₋ RESET is driven by aninternal event, generated by state ST₋₋ IDLE state processing.

Although the preferred embodiment uses only one state to perform certainstate processing, that state processing may be equivalently performed inmultiple states. Likewise, the state processing of multiple states ofthe described embodiment may be equivalently performed in a singlestate.

Although the prototype uses a state table to control the flow of programexecution and to select one of a plurality of selectable regions, thesame function may be equivalently performed using object orientedsoftware architecture, if-then-else statements or a combination ofthese. In particular, in accord with object oriented softwarearchitecture, each state machine may be equivalently represented as aninstantiation of a selectable region class for processing inputsaffecting a particular selectable region.

Although the described embodiment uses a single processor, state tableand code for state processing for all the selectable regions, each ofthese may be duplicated. Alternate embodiments may include processors,electronic circuitry, state tables or code for state processing used forprocessing fewer than all the selectable regions or used to processcertain selectable regions at one time and other selectable regions atother times.

In the prototype, each state machine processes events independently ofall other state machines, though a state machine may send an event toanother state machine. For example, when the operator selects aselectable region, the associated state machine sends the event EV₋₋RESET to all other state machines so that all selectable regions revertto their respective initial colors.

A single operator action may result in the issuance of different eventsto different state machines. For example, when the operator moves thecursor from without a selectable region to within it, EV₋₋ DWELL is sentto the newly indicated state machine. EV₋₋ MOVEMENT is sent to all otherstate machines.

Following is a general description of each event used in the prototypeand an example of the use of each event. For all uses of each event,refer to the state table shown in FIGS. 29 and 30 which determines whatstate transition occurs from every state on occurrence of a given event.

The event EV₋₋ RESET is an internal event which drives a state machineto its initial state. For example, when the operator has not moved thepointer (2202) for a predetermined period of time, event EV₋₋ RESET issent to all other selectable regions.

The event EV₋₋ DECAY is an external event which indicates that thecursor hotspot does not intersect the selectable region associated withthe state machine. EV₋₋ DECAY is sent to a state machine periodicallywhen the operator has positioned the cursor hotspot on a selectableregion other than the selectable region associated with that statemachine.

The event EV₋₋ DWELL is an external event which indicates that thecursor hotspot intersects the selectable region associated with thestate machine. EV₋₋ DWELL is sent to a state machine periodically whenthe operator has positioned the cursor hotspot on the associatedselectable region.

The event EV₋₋ CROSS₋₋ OUT is an external event which indicates that thecursor hotspot has moved from a location intersecting the selectableregion associated with the state machine to a location not intersectingthe selectable region. After the operator selects a selectable region,he must move the cursor hotspot out of the selectable region, generatingEV₋₋ CROSS₋₋ OUT, before he can again select that selectable region.

The event EV₋₋ STEP₋₋ UP is an internal event which indicates that aselectable region's selection threshold has been satisfied.

The event EV₋₋ MOVEMENT is an external event which indicates that thecursor hotspot has moved. If the cursor hotspot intersects a selectableregion without moving for a predetermined period of time, a timeoutoccurs, causing all state machines to transition to the reset state.EV₋₋ MOVEMENT drives the state machine out of the reset state.

The event EV₋₋ IDLE₋₋ TIMEOUT is an external event which indicates thatthe cursor hotspot has intersected a selectable region without movingfor a predetermined period of time. EV₋₋ IDLE₋₋ TIMEOUT causes the statemachine to move the cursor hotspot to the center of the display.

The event EV₋₋ CEILING is an external event which indicates that thecursor hotspot intersects a selectable region and the color of theselectable region equals the selectable region color ceiling. If thelocking feature is enabled, EV₋₋ CEILING drives the state machine to thebegin lock state where it displays the lock icon.

The null event, EV₋₋ NULL, is a multi-purpose internal event used in avariety of situations to drive a state machine to another state. Forexample, after a timeout has been detected, EV₋₋ IDLE₋₋ TIMEOUT isgenerated and sent to the appropriate state machine driving it to theidle state, the receiving state machine sends itself EV₋₋ NULL in orderto drive itself to the reset state. The use of EV₋₋ NULL here allowsstates to be simpler and the reset state to be reused.

The prototype uses eight partially delimited selectable regions. In thedescription below, the portion of each selectable region shown on thedisplay is referred to as the visible subregion of the selectableregion. The portion of each selectable region outside the display isreferred to as the invisible subregion of the selectable region. Becausethe software driver (1202) confines the cursor hotspot to the Windows®cursor clipping rectangle, a rectangle on the display slightly smallerin area than the display, the access program (1206) of the prototypeonly reads hotspot cursor locations within the Windows® cursor clippingrectangle, even though the operator may in fact be pointing to alocation outside the Windows® cursor clipping rectangle, e.g. within aninvisible subregion. Thus the access program (1206) does not distinguishbetween two locations indicated by the operator, the first at a firstlocation on the edge of the Windows® cursor clipping rectangle and thesecond outside the Windows® cursor clipping rectangle whose location isreported by the software driver (1202) to be the first location. Forexample, assuming that the operator moves the location indicated by thepointer (2202) to a location within invisible subregion (0104 in FIG.17), the software driver (1202) in the prototype reports the cursorhotspot location to be the closest point within the visible subregion(0106). Consequently, in the prototype, all invisible subregions areunbounded on their side furthest from and parallel to the edge of thedisplay. Thus, in the prototype, point (0162) in FIG. 17 lies withinselectable region (0128/0130) since the rightmost side of selectableregion (0128/0130) is unbounded.

The operation of the prototype will now be described using, as anexample, the selection of a menu option associated with selectableregion (0104/0106). First described are notation conventions used in thedescription, then initialization in the prototype, and then the example.The description refers to the procedures PocketFsm and CreateEvent.These procedures are listed in pseudo-code in Appendix I.

Notational conventions used in the description below:

1. pPocket→indicates a set of data associated with a particular statemachine, in this example, the state machine associated with selectableregion (0104/0106).

2. pPocket→State indicates a particular item of data within the set ofdata associated with the state machine, in this case, the variable"State".

During initialization:

1. The Windows® cursor clipping rectangle is set so that most of thearrow cursor is always visible on the display.

2. The cursor, indicating on the display (2112) the location indicatedby the movement related signal receiving means, is positioned at thecenter of the display.

3. All data associated with each state machine are initialized. Thefollowing variables are included in the set of data associated with eachstate machine. There are as many independent copies of these variablesas there are state machines. For each state machine the followingvariables are initialized as indicated.

    __________________________________________________________________________    variable  initial value    meaning                                            __________________________________________________________________________    fInvert   FALSE            if TRUE, display the color complementary to                                   that                                                                          indicated by this selectable region's Color                                   variable                                           State     ST.sub.-- INITIAL                                                                              state of the state machine                         PreviousState                                                                           0                previous state of the state machine                Color     0                RGB encoding of visible subregion color            fPaint    FALSE            if TRUE, paint visible subregion on receipt                                   of                                                                            WM.sub.-- PAINT message                            InitialColor                                                                            RGB (0,32,0)     initial value corresponds to a dark green          pLabel    initial menu option                                                                            indicates an element within the                              for the state machine's                                                                        aLabel array, described beIow                                selectable region                                                   Decrement RGB (0,1,0)      value subtracted from Color on EV.sub.--                                      DECAY                                              Increment RGB (0,4,0)      value added to Color on EV.sub.-- DWELL            Ceiling   RGB (0,255,0)    initial value corresponds to a very light,                                    bright green                                       CrestTide RGB (0,143,0)    initial value corresponds to a                                                light green                                        Lockspot.x                                                                              a point within the                                                                             x coordinate of location for the                             selectable region                                                                              display ofthe lock icon                                      located two thirds of                                                         the length of the                                                             selectable region from                                                        the closest corner of                                                         the display                                                         Lockspot.y                                                                              a point within the                                                                             y coordinate of location for the                             selectable region                                                                              display of the lock icon                                     located two thirds of                                                         the length of the                                                             selectable region from                                                        the closest corner of                                                         the display                                                         fInterior TRUE for interior                                                                              if TRUE, this is an interior region                          regions, otherwise                                                                             of adjacent regions                                          FALSE; used for                                                               the Intersection                                                              aspect of the invention                                             iAdjacentPocket                                                                         index of state machine associated                                             with the adjacent region; used for                                            the Intersection                                                              aspect of the invention                                             first array of points                                                                   boundaries of the                                                             visible subregion                                                   second array of points                                                                  boundaries of the                                                             region; used for the Intersection                                             aspect of the invention                                             hRegion   handle to Windows ® region                                                corresponding to the visible subregion;                                       used for the Intersection                                                     aspect of the invention                                             __________________________________________________________________________

4. aLabel, an array of data structures defining the menu and submenuoptions and the menu hierarchy, is initialized. For example, one of theelements of aLabel defines menu option "vort<space>x". This elementincludes fields which determine that this menu option is displayedhorizontally starting at certain (x,y) coordinates, that on selectioncertain actions are to be taken, for example, outputting text to aspeech synthesizer, and that on selection certain submenu options, inthis example, "v", "o", "r", "t", "<space>" and "x", and related dataare to be associated with certain state machines. In the prototype, thisassociation is accomplished by modifying pLabel in the set of data ofthe associated state machine to point to the aLabel elementcorresponding to the menu option to be associated with that statemachine.

5. At least one window is created in the circumscribed region (0150) forthe display of selected letters.

6. The state table is initialized to the values shown in FIGS. 29 and30.

7. EV₋₋ RESET is sent to each state machine by a procedure call of theform PocketFsm (pPocket, EV₋₋ RESET). As an example, assume pPocketindicates the state machine associated with selectable region(0104/0106). Proceeding through the pseudo-code for the procedurePocketFsm listed in Appendix I, fInternalEvent is set to TRUE andconsequently control passes into the while loop. fInternalEvent is nowset to FALSE. The current state, pPocket→State, is stored inpPocket→PreviousState. Now a state transition is made. The currentstate, ST₋₋ INITIAL, having a value of 2, and the current event, EV₋₋RESET, having a value of 1, are used as row and column indicesrespectively into the state table aPocketFsm shown in FIGS. 29 and 30,to determine the value of the new state of the state machine, in thisexample, the state machine associated with selectable region(0104/0106). aPocketFsm[2][1] equals 3. Thus the new state of the statemachine is 3, the value of ST₋₋ RESET. Control passes, via the switchstatement, to the ST₋₋ RESET case and ST₋₋ RESET state processing isperformed. The time of selectable region selection is set to the currenttime, the state machine's fInvert flag is set to FALSE and the value ofthe state machine's color variable, pPocket→Color, is compared to thestate machine's initial color, pPocket→InitialColor. pPocket→Color wasinitialized to zero, which is not the value of pPocket→InitialColor.Consequently, pPocket→Color is set to pPocket→InitialColor and the flagpPocket→fPaint is set to TRUE. Upon reaching the break statement,control passes through the bottom of the switch statement and the valueof pPocket→fPaint is tested. Since it is TRUE, the client area rectangleis invalidated. In the prototype, Windows® (1204) responds toinvalidating the client rectangle by sending the access program (1206) aWM₋₋ PAINT message. On receipt of a WM₋₋ PAINT message, the accessprogram (1206) redraws all selectable regions and any menu optionlocated thereon for each state machines having pPocket→fPaint equal toTRUE. Thus, somewhat indirectly, visible subregion (0106) is drawn onthe display (2112). Then control returns to Windows® (1204). The otherseven state machines, each associated respectively with a selectableregion, are similarly initialized so that each state machine transitionsfrom ST₋₋ INITIAL to ST₋₋ RESET and draws its respective visiblesubregion and any menu option located thereon on the display (2112). Thedisplay shown in FIG. 17 now appears on the display (2112) of thecomputer system (2116).

8. A periodic timer, called the cursor polling timer, is set. This timerprovides the access program (1206) with a WM₋₋ TIMER message at frequentintervals, in the prototype every system clock tick which occursapproximately every 54 milliseconds. Following access program (1206)initialization, most state transitions are made on expiration of thecursor polling timer. The access program (1206) calls the procedureCreateEvent to determine the appropriate event for each state machineand to complete the event data structure addressed by pEventaccordingly, then repetitively calls the procedure PocketFsm for eachstate machine, passing the unique indicator for the state machine andthe appropriate event for that state machine. Preferably, the cursorpolling timer is more frequent so that color changes to visiblesubregions are smaller and more frequent, giving a smoother appearanceto color change.

An example of the selection of selectable region (0104/0106) in accordwith the Perimeter Menu aspect of the invention will now be described.Following initialization, assume that the operator now begins to movethe pointer (2202). Every 54 milliseconds the cursor polling timerexpires, causing Windows® (1204) to send a WM₋₋ TIMER message to theaccess program (1206). Following receipt of WM₋₋ TIMER, the accessprogram (1206) calls the procedure CreateEvent. The procedureCreateEvent, among other functions, determines whether the currentcursor hotspot location lies within any selectable region. In thisexample, the operator is moving the cursor from its initial location inthe center of the display toward selectable region (0104/0106), butsince only 54 milliseconds have elapsed, the cursor hotspot has movedonly slightly in that direction. The procedure CreateEvent determinesthat the cursor hotspot does not lie within any selectable region andthat the cursor hotspot has not crossed out of a selectable region inthe past 54 milliseconds. Therefore, the procedure CreateEventdetermines that each state machine should receive EV₋₋ MOVEMENT. Theprocedure PocketFsm is called with the indicator for the state machineassociated with selectable region (0104/0106) and with EV₋₋ MOVEMENT.The event EV₋₋ MOVEMENT drives this state machine from its currentstate, ST₋₋ RESET, to ST₋₋ EBB₋₋ TIDE. The pseudo-code for ST₋₋ EBB₋₋TIDE in procedure PocketFsm is a break statement, indicating that nostate specific action is taken at this time, other than the transitionto ST₋₋ EBB₋₋ TIDE. Control returns to Windows® (1204).

Shortly before or shortly after the state machine associated withselectable region (0104/0106) receives EV₋₋ MOVEMENT, all other statemachines each receive EV₋₋ MOVEMENT and each makes the same transitionfrom ST₋₋ RESET to ST₋₋ EBB₋₋ TIDE.

Another 54 milliseconds elapses and again procedure PocketFsm is called,sending EV₋₋ MOVEMENT to the state machine associated with selectableregion (0104/0106) and driving state machine from ST₋₋ EBB₋₋ TIDE toST₋₋ DECAY. Stepping through the pseudo-code for ST₋₋ DECAY, the ST₋₋DECAY state sets pPocket→State to the value stored inpPocket→PreviousState and decrements pPocket→Color but not below thevalue of pPocket→InitialColor. pPocket→Color determines the color andbrightness of visible subregion (0106). Decrementing pPocket→Colorresults in a darkening of visible subregion (0106). Resuming with thepseudo-code for ST₋₋ DECAY state processing, the ST₋₋ DECAY state setsfInternalEvent to TRUE, and, in this case, sets Event to EV₋₋ NULL.Following the break statement, the while fInternalEvent condition istrue and another state transition occurs, using the value ofpPocket→State which was set by ST₋₋ DECAY state processing to ST₋₋ EBB₋₋TIDE, the previous state. The new state is found at aPocketFsm[ST₋₋EBB₋₋ TIDE][EV₋₋ NULL], which equals ST₋₋ EBB₋₋ TIDE. This statetransition is unlike an ordinary state transition because the startingstate is set by ST₋₋ DECAY. All transitions from ST₋₋ DECAY share thisdistinction. The state machine executes the code for the new state, ST₋₋EBB₋₋ TIDE, which is simply a break statement. The procedure PocketFsmdetermines that fpaint is FALSE and exits. Control returns to Windows®(1204).

Shortly before or shortly after the state machine associated withselectable region (0104/0106) receives EV₋₋ MOVEMENT, all other statemachines each receive EV₋₋ MOVEMENT and each makes the same transitionsfrom ST₋₋ EBB₋₋ TIDE to ST₋₋ DECAY to ST₋₋ EBB₋₋ TIDE.

Every 54 milliseconds this scenario is repeated for each state machineuntil the operator moves the cursor hotspot to a point within selectableregion (0104/0106). At this time the procedure CreateEvent determinesthat there is an active selectable region, specifically selectableregion (0104/0106), and that consequently EV₋₋ DWELL should be sent tothe associated state machine. The procedure PocketFsm is called with theindicator for the state machine associated with selectable region(0104/0106) and the event EV₋₋ DWELL. EV₋₋ DWELL drives this statemachine from ST₋₋ EBB₋₋ TIDE to ST₋₋ ENTRY. Following the pseudo-codefor ST₋₋ ENTRY state processing shown in PocketFsm pseudo-code,fInternalEvent is set to TRUE and Event is set to EV₋₋ NULL, resultingin another state transition to aPocketFsm[ST₋₋ ENTRY][EV₋₋ NULL], whichequals ST₋₋ LOW₋₋ TIDE.

The pseudo-code for ST₋₋ LOW₋₋ TIDE is only a break statement, so thereis no state specific action for ST₋₋ LOW₋₋ TIDE other than entry intothis state. fpaint is FALSE so the selectable region is not redrawn.Control returns to Windows® (1204).

Shortly before or shortly after the state machine associated withselectable region (0104/0106) receives EV₋₋ DWELL, all other statemachines each receive EV₋₋ MOVEMENT and each makes the same transitionsfrom ST₋₋ EBB₋₋ TIDE to ST₋₋ DECAY to ST₋₋ EBB₋₋ TIDE.

Another 54 milliseconds elapses. The procedure CreateEvent determinesthat the state machine associated with selectable region (0104/0106)should receive EV₋₋ MOVEMENT, which drives it from ST₋₋ LOW₋₋ TIDE toST₋₋ DWELL. Stepping through the pseudo-code for ST₋₋ DWELL, the ST₋₋DWELL state sets pPocket→State to the value stored inpPocket→PreviousState, increments pPocket→Color by pPocket→Increment,but not above the value of pPocket→Ceiling, sets pPocket→fPaint to TRUE,sets fInternalEvent to TRUE, and, in this case, sets Event to EV₋₋ NULL.Following the break statement, the while fInternalEvent condition istrue and another state transition occurs. The new state is found ataPocketFsm[ST₋₋ LOW₋₋ TIDE][EV₋₋ NULL], which equals ST₋₋ LOW₋₋ TIDE.This path is unlike an ordinary state transition because the startingstate is set by ST₋₋ DWELL. All transitions from ST₋₋ DWELL share thisdistinction. The state machine executes the code for the new state, ST₋₋LOW₋₋ TIDE, which is simply a break statement. The procedure PocketFsmdetermines that fpaint is TRUE, invalidates the client rectangle andexits. As a result of invalidating the client rectangle, Windows® (1204)sends a WM₋₋ PAINT message to the access program (1206). On receipt ofWM₋₋ PAINT, the access program (1204) checks the value of fpaint foreach state machine, and if TRUE, sets fpaint to FALSE and redraws thevisible subregion of the selectable region associated with that statemachine and any menu option located thereon. The color of the redrawnvisible subregion is determined by the value of the Color variable forthat state machine. After redrawing, control returns to Windows® (1204).

Shortly before or shortly after the state machine associated withselectable region (0104/0106) receives EV₋₋ DWELL, all other statemachines each receive EV₋₋ MOVEMENT and each makes the state transitionsfrom ST₋₋ EBB₋₋ TIDE to ST₋₋ DECAY to ST₋₋ EBB₋₋ TIDE.

Assuming the operator maintains the cursor hotspot in the selectableregion (0104/0106), the state machine associated with selectable region(0104/0106) cycles repetitively through the state transitions from ST₋₋LOW₋₋ TIDE to ST₋₋ DWELL to ST₋₋ LOW₋₋ TIDE, driven by the cursorpolling timer. With each transition to ST₋₋ DWELL, visible subregion(0106) is brightened a bit. The polling timer interval is short enoughand the increment to pPocket→Color is small enough that visiblesubregion (0106) appears to gradually brighten although in fact itprogresses rapidly through a series of discrete brightness levels. Witheach iteration through ST₋₋ DWELL, pPocket→Color in incremented.

Assuming that the operator maintains the cursor hotspot on selectableregion (0104/0106), pPocket→Color will eventually equal or exceedpPocket→CrestTide, a variable set at initialization time and not changedthereafter. At this time, Event is set to EV₋₋ STEP₋₋ UP, driving atransition to aPocketFsm[ST₋₋ LOW₋₋ TIDE][EV₋₋ STEP₋₋ UP], which equalsST₋₋ SELECTED. The state processing for ST₋₋ SELECTED provides theoperator with an audible indication that a selection has just been made,takes the action appropriate upon selection of this selectable region,including selecting the menu option associated with the selectedselectable region. In this example, the access program (1206) does notgenerate output to another program or device at this time. ST₋₋ SELECTEDnext sets pPocket→fInvert to TRUE, and, if appropriate, changes the menuoptions associated with various selectable regions. In this example,selectable region (0104/0106) is now associated with menu option"<space>", selectable region (0108/0110) with menu option "o",selectable region (0112/0114) with menu option "t", selectable region(0124/0126) with menu option "x", selectable region (0128/0130) withmenu option "v", and selectable region (0132/0134) with menu option "r".Selectable regions (0116/0118) and (0120/0122) remains associated withthe same menu options with which they were associated in FIG. 17. fpaintis set to TRUE for the state machines associated with the selectableregions having changed menu options. ST₋₋ SELECTED next sets Event toEV₋₋ NULL and fInternalEvent to TRUE. The state machine now makes thetransition to aPocketFsm[ST₋₋ SELECTED₋₋ TIDE][EV₋₋ NULL], which equalsST₋₋ CREST₋₋ TIDE. There is no state specific action for ST₋₋ CREST₋₋TIDE other than entry into this state. The pseudo-code for ST₋₋ CREST₋₋TIDE is only a break statement. Control returns to Windows® (1204).

When the access program (1206) later receives the WM₋₋ PAINT messageresulting from the invalidating of the client rectangle, the setting offInvert to TRUE causes the visible subregion (0106) to be drawn in thecolor complementary to the value then indicated by pPocket→Color. Thesetting of fpaint to TRUE for all selectable regions associated withchanged menu options causes those selectable regions and the menuoptions thereon to be redrawn. The display shown in FIG. 18 now appearson the display (2112) of the computer system (2116).

Assuming the operator maintains the cursor hotspot in the selectableregion (0104/0106), the state machine associated with selectable region(0104/0106) cycles repetitively through the state transitions from ST₋₋CREST₋₋ TIDE to ST₋₋ DWELL to ST₋₋ CREST₋₋ TIDE, driven by the cursorpolling timer. With each transition to ST₋₋ DWELL, visible subregion(0106) is brightened a bit, though it is now magenta, the complement ofgreen. The polling timer interval is short enough and the increment topPocket→Color is small enough that visible subregion (0106) appears togradually brighten although in fact it progresses rapidly through aseries of discrete brightness levels. With each iteration through ST₋₋DWELL, pPocket→Color in incremented. Assuming that the operatormaintains the cursor hotspot on selectable region (0104/0106),pPocket→Color will eventually equal pPocket→Ceiling, a variable set atinitialization time and not changed thereafter.

Assuming that the operator now moves the cursor so that the hotspot islocated outside selectable region (0104/0106), CreateEvent generates theEV₋₋ CROSS₋₋ OUT for the state machine associated with selectable region(0104/0106), driving the state machine to aPocketFsm[ST₋₋ CREST₋₋TIDE][EV₋₋ CROSS₋₋ OUT], which equals ST₋₋ SELECT₋₋ AND₋₋ OUT. The stateprocessing for ST₋₋ SELECT₋₋ AND₋₋ OUT sends EV₋₋ RESET and then EV₋₋MOVEMENT to all state machines, driving each of them from their currentstate to ST₋₋ RESET and then to ST₋₋ EBB₋₋ TIDE. For each state machine,the transition ST₋₋ RESET and then to ST₋₋ EBB₋₋ TIDE results in actionspreviously described for those states, except that, unlike before, thevalue of pPocket→Color for the state machine associated with selectableregion (0104/0106) does not equal the value of pPocket→InitialColor forthe state machine associated with selectable region (0104/0106).Consequently, pPocket→Color is set to pPocket→InitialColor and fpaint isset to TRUE, subsequently causing visible subregion (0106) to be redrawnin its initial color. Control returns to Windows® (1204).

The prototype continues to sample cursor location at 54 millisecondintervals, determine the appropriate event for each state machine andsend that event each state machine, causing state transitions in eachstate machine according to the state table shown in FIGS. 29 and 30.Assuming that the operator next selects selectable region (0108/0110),the state processing in ST₋₋ SELECTED displays the selection, the letter"o", in the circumscribed region (0150), and associates the menu optionsshown in FIG. 17 with their respective selectable regions. The displayshown in FIG. 17 appears on the display (2112) of the computersystem.(2116).

B. Confinement

The preferred embodiment of the Confinement aspect of the invention willnow be described in detail from a functional perspective using anexample illustrated in FIG. 31. In FIG. 31, 20 selectable regions, e.g.(1704), (1708) and (1714), are depicted on display (2112). Each of theselectable regions is located on the display (2112) adjacent an edge ofthe display, and the selectable regions together circumscribe a region(1702) on the display. The top and left edges of the Windows® cursorclipping rectangle (1750) lie on the top and left edges, respectively,of the display (2112). The bottom and right edges of the Windows® cursorclipping rectangle (1750) lie on the display parallel to and slightlyindented from the bottom and right edges, respectively, of the display.A confining polygon is delimited on the display. The boundary of theconfining polygon, starting from the upper right corner of the Windows®cursor clipping rectangle, follows the top edge of the Windows® cursorclipping rectangle to the left until it reaches region (1732), where theboundary follows the side of region (1732) down, to the left, and backup to the top edge of the Windows® cursor clipping rectangle. Theboundary continues left along the top edge of the Windows® cursorclipping rectangle until it reaches region (1734), where the boundaryfollows the side of region (1734) down, to the left, and back up to thetop edge of the Windows® cursor clipping rectangle. The boundarycontinues left along the top edge of the Windows® cursor clippingrectangle to the upper left corner and then turns down along the leftedge of the Windows® cursor clipping rectangle until it reaches region(1736), where the boundary follows the side of region (1736) to theright, down, and to the left to the left edge of the Windows® cursorclipping rectangle. The boundary of the confining polygon continues inthis fashion around to the upper right corner of the Windows® cursorclipping rectangle. The confining polygon thus includes all the area ofthe Windows® cursor clipping rectangle except for the regions (1732),(1734), (1736), (1738), (1740), (1742), (1744), and (1746). An operatorcontrolling a pointer indicating successive locations with respect tothe display and attempting to select a target selectable region mayovershoot the target so that some of the successive locations lieoutside the Windows® cursor clipping rectangle. In the preferredembodiment of the Confinement aspect of the invention, the cursor (1724)is confined to the confining polygon. The preferred embodiment isresponsive to an intersection of the cursor hotspot and any oneselectable region so that an overshot selectable region may be selectedby click or by dwell without moving the location presently indicated bythe pointer to a location in the overshot selectable region. Thus, thepreferred embodiment of the Confinement aspect of the invention allowsan operator with impaired ability to stop motion to maintain the cursormore easily on a selectable region, and so select the intendedselectable region, than do conventional user interface systems.

A selectable region having a single side abutting a confining polygonprevents cursor movement only beyond the abutting side. However, NMDoperators who drift may drift in more than one direction. Assume that acertain NMD operator tends to drift both up and to the left and that heis attempting to select selectable region (1704). If he moves the cursorinto that selectable region his upward drift will be confined; the driftwill not move the cursor beyond the confining polygon. However, thecursor will move to the left, since, in the preferred embodiment of theConfinement aspect of the invention, movement in this direction is notaffected by the confining polygon, and consequently the cursor may moveinto selectable region (1708), the selectable region to the left ofselectable region (1704). NMD operators having this type of drift may beassisted in selecting by confining corners. For example, such anoperator, attempting to select selectable region (1704) could move thecursor to location (1706) in selectable region (1704). As the operatordrifts to the left, he can compensate by moving the pointer to theright. Assuming the operator lacks fine motor control, he mayovercompensate and indicate a location to the right of theWindows(cursor clipping rectangle (1750). However, since the cursor isconfined to the confining polygon, the cursor remains in the intendedselectable region.

Confining corners facilitate the selection process for some NMDoperators. The preferred embodiment of the Confinement aspect of theinvention creates a corner or virtual corner in each selectable region.A virtual corner is a corner of a selectable region formed by theintersection of two sides of a selectable region both of which abut aconfining polygon. For example, corner (1710) in selectable region(1708) abuts the confining polygon both along the top edge of theWindows® cursor clipping rectangle and along the right side of region(1732). If an NMD operator drifts from selectable region (1708) to theleft into region (1732), the cursor remains in selectable region (1708).Thus, drift to the left does not move the cursor out of selectableregion (1732). An operator trying to select selectable region (1708) mayovercompensate for drift to the right by moving the pointer to indicatea location in region (1732).

FIG. 32 illustrates a display in accordance with an alternativeembodiment of the Confinement aspect of the invention. FIG. 32 depicts16 selectable regions, e.g. selectable region (0602), on a display(2112), the selectable regions together at least partiallycircumscribing region (0660) on the display. Circumscribed region (0660)intersects four selectable regions, (0604), (0608), (0610), and (0612).Selectable region (0608) includes virtual corner (0652). The confiningpolygon includes all the area of the Windows® cursor clipping rectangleexcept for regions (0632), (0634), (0636), (0638), (0640), (0642),(0644), and (0646).

FIG. 33 illustrates a display and structures in accordance with ananother embodiment of the Confinement aspect of the invention. FIG. 33depicts 20 selectable regions, e.g. selectable region (0918), on adisplay (2112), the selectable regions together at least partiallycircumscribing region (0902) on the display. The confining polygonincludes all the area of the Windows® cursor clipping rectangle exceptfor regions (0906), (0908), (0910), and (0904).

The preferred embodiment of the Confinement aspect of the invention willnow be described in detail from an implementation perspective.Preferably, the Confinement aspect is implemented by modifications tothe access program (1206) described in the detailed description of thePerimeter Menu aspect of the invention. The modifications required are:(1) define 20 state machines and 20 respectively associated selectableregions located as shown in FIG. 31; (2) at initialization time, createa Windows® region corresponding to the confining polygon described abovein the functional description of the preferred embodiment of theConfinement aspect of the invention; and (3) modify the procedureCreateEvent so that before generating an EV₋₋ CROSS₋₋ OUT event,CreateEvent determines whether the current hotspot cursor locationintersects the confining polygon, and, if not, set the Windows® hotspotcursor location to the previous hotspot cursor location and transferscontrol to the code at the beginning of the CreateEvent procedure whichgets the current cursor hotspot location from Windows®. If the currenthotspot cursor location intersects the confining polygon, CreateEventtakes the same action as in the access program (1206).

C. Dwell

The preferred embodiment of the Dwell aspect of the invention will nowbe described in detail from a functional and implementation perspective.The prototype implements dynamic dwell. The effect of dwelling on aselectable region and its implementation have been described in thedetailed description of the Perimeter Menu aspect of the invention. Nowthe effect and implementation of moving the cursor hotspot off aselectable region will be described.

Referring now to FIG. 17, assuming that the state machine associatedwith the selectable region (0104/0106) is in state ST₋₋ LOW₋₋ TIDE, thatpPocket→Color has been incremented above its initial value, that thecursor hotspot intersected selectable region (0104/0106) al the lastexpiration of the cursor polling timer, and that the operator initiatesmovement of the cursor hotspot to the right from selectable region(0104/0106) toward selectable region (0132/0134) so that, when thecursor polling timer next expires, the cursor hotspot is located in area(0150) between visible subregions (0106) and (0134), the procedureCreateEvent determines that the state machine associated with selectableregion (0104/0106) should receive EV₋₋ CROSS₋₋ OUT, which drives it fromST₋₋ LOW₋₋ TIDE to ST₋₋ LOW₋₋ TIDE. The state processing for ST₋₋ LOW₋₋TIDE has already been described in the detailed description of thePerimeter Menu aspect of the invention. Assuming that the operatorcontinues to move the cursor hotspot toward selectable region(0132/0134), on the next expiration of the cursor polling timer, theprocedure CreateEvent determines that the state machine associated withselectable region (0104/0106) should receive EV₋₋ MOVEMENT, which drivesit from ST₋₋ LOW₋₋ TIDE to ST₋₋ DECAY. Stepping through the pseudo-codefor ST₋₋ DECAY, the ST₋₋ DECAY state sets pPocket→State to the valuestored in pPocket→PreviousState, decrements pPocket→Color bypPocket→Decrement, but not below the value of pPocket→InitialColor, setspPocket→fPaint to TRUE, sets fInternalEvent to TRUE, and, in this case,sets Event to EV₋₋ NULL. Following the break statement, the whilefInternalEvent condition is true and another state transition occurs.The new state is found at aPocketFsm[ST₋₋ LOW₋₋ TIDE][EV₋₋ NULL], whichequals ST₋₋ LOW₋₋ TIDE. This path is unlike an ordinary state transitionbecause the starting state is set by ST₋₋ DECAY. All transitions fromST₋₋ DECAY share this distinction. The state machine executes the codefor the new state, ST₋₋ LOW₋₋ TIDE, which is simply a break statement.The procedure PocketFsm determines that fPaint is TRUE, invalidates theclient rectangle and exits. As a result of invalidating the clientrectangle, Windows® (1204) sends a WM₋₋ PAINT message to the accessprogram (1206). On receipt of WM₋₋ PAINT, the access program (1204)checks the value of fPaint for each state machine, and if TRUE, setsfpaint to FALSE and redraws the visible subregion of the selectableregion associated with that state machine and any menu option locatedthereon. The color of the redrawn visible subregion is determined by thevalue of the Color variable for that state machine. After redrawing,control returns to Windows® (1204).

If the movement of the cursor hotspot pause between successive samplingsof its location, the procedure Create Event will determine that not EV₋₋MOVEMENT, but EV₋₋ DECAY, should be sent to the state machine associatedwith selectable region (0104/0106). Like EV₋₋ DECAY, EV₋₋ MOVEMENTdrives the state machine associated with selectable region (0104/0106)to ST₋₋ DECAY. The same state processing as described above for ST₋₋DECAY takes place, including the transition back to ST₋₋ LOW₋₋ TIDE.Assuming the operator maintains the cursor hotspot in area (0150), thestate machine associated with selectable region (0104/0106) cyclesrepetitively through the state transitions from ST₋₋ LOW₋₋ TIDE to ST₋₋DECAY to ST₋₋ LOW₋₋ TIDE, driven by the cursor polling timer. With eachtransition to ST₋₋ DECAY, visible subregion (0106) is darkened a bit.The polling timer interval is short enough and the decrement topPocket→Color is small enough that visible subregion (0106) appears togradually darken although in fact it progresses rapidly through a seriesof discrete brightness levels. With each iteration through ST₋₋ DECAY,pPocket→Color in decremented until pPocket→Color reachespPocket→InitialColor.

When the cursor hotspot reaches selectable region (0132/0134), theprocedure CreateEvent determines that the state machine associated withselectable region (0104/0106) should receive EV₋₋ DECAY, continuing therepetitively cycling through from ST₋₋ LOW₋₋ TIDE to ST₋₋ DECAY to ST₋₋LOW₋₋ TIDE, driven by the cursor polling timer. Driven by the samepolling timer, the procedure CreateEvent determines that the statemachine associated with selectable region (0132/0134) should receiveEV₋₋ DWELL, driving this state machine through the state transitionsdescribed in the detailed description of the Perimeter Menu aspect ofthe invention.

If the operator moves the cursor hotspot from selectable region(0132/0134) back to selectable region (0104/0106), the procedureCreateEvent determines that the state machine associated with selectableregion (0104/0106) should receive EV₋₋ DWELL, driving this state machinethrough the state transitions described in the detailed description ofthe Perimeter Menu aspect of the invention.

In the prototype, the selectable regions are shown on a display (2112).Alternatively, the selectable regions may appear on a static display, orthey may be projected on a surface.

In the prototype, the brightness of a visible subregion at any timeindicates the progress of the selection of the selectable regionincluding the visible subregion. A brightness close to the initialbrightness indicates that a relatively long period of dwelling on thisselectable region is required for selection. A brightness close to thebrightness just prior to selection indicates that relatively shortperiod of dwelling on this selectable region is required for selection.However, means for indicating an intersection of the location indicatedby the movement related signal and a selectable region, or the durationof a period of such an intersection includes, but is not limited to, achange in cursor appearance or location, a change in location, size,shape, hue, brightness, contrast, tone, dithering, pattern, hatching,font or fill of an object on the surface, a display of or change in agraphic on the surface or the removal of a graphic from the surface, ageneration of a sound or a change in the pitch or volume of an extantsound, a change in the temperature or surface of a contact area, thepressure exerted by a contact area, or frequency of contact by a contactarea, or other suitable means. Any of these indications may becontinuous or frequent.

Although dwell is implemented in the prototype using a data value,specifically pPocket→Color, it may be implemented using a signal, forexample, voltage or current, varying in response to the intersection andsubsequent non-intersection of a location indicated by a pointer and aselectable region. For example, a selectable region may include adetector and coupled electronics or electrical circuitry operative toincrease the voltage level of a capacitor. Once elevated, the voltagelevel may decrease over time. Upon reaching a predetermined threshold,the voltage level may trigger selection.

The prototype allows an operator to make selections by dwell moreefficiently than in conventional systems. In the prototype, thebrightness of a visible subregion indicates the dwell time required forselection. A practiced operator may accurately estimate when he may planhis next pointer movement, when he may begin moving the pointer and maydetermine when a bit more exertion will select a selectable region andwhen it will not. Thus a disabled operator who is fatigued by computeraccess and can maintain a pointer in a steady position for only briefperiods, may optimize his energy expenditure, for example, exertinghimself to maintain the cursor on a certain selectable region only whendoing so will quickly select the selectable region.

The prototype may increase the independence of a disable individual byallowing him to control devices such as a TV, thermostat and otherhousehold appliances. As stated earlier, the Perimeter Menu aspect ofthe invention may be implemented on a general purpose computer system.If the general purpose computer system is coupled to a devices capableof executing commands and the menu hierarchy allows the selection ofcommands, a disabled operator may select and issue commands to controlthese devices.

FIG. 34 illustrates a display in accordance with of an embodiment of theDwell aspect of the invention having 20 selectable regions (3101),(3103), (3105), (3107), (3109), (3111), (3115), (3117), (3119), (3121),(3125), (3127), (3129), (3131), (3133), (3135), (3139), (3141), (3143),and (3145) circumscribing a surface (3151) having an indicating region(3147) thereon.

FIG. 35 illustrates an apparatus in accordance with of an embodiment ofthe Dwell aspect of the invention having 20 selectable regionscircumscribing a surface, each selectable region associated respectivelywith an indicating region adjacent its associated selectable region.Selectable region (3101) is associated with indicating region (3201),selectable region (3103) with indicating region (3203), etc.

FIG. 36 illustrates an apparatus in accordance with of an embodiment ofthe Dwell aspect of the invention having 20 detectors circumscribing anaperture (3350), each detector associated respectively with an indicatorintersecting its associated detector, e.g. detector (3301) is associatedwith indicator (3303).

FIG. 37 illustrates an apparatus in accordance with of an embodiment ofthe Dwell aspect of the invention having 24 selectable regions arrangedin a grid of four rows and six columns, each selectable regionassociated respectively with an indicating region intersecting itsassociated selectable region. For example, selectable region (3402) isassociated with indicating region (3401).

FIG. 38 illustrates a display in accordance with of an embodiment of theDwell aspect of the invention having a plurality of detectors, (2501),(2503), (2505), etc., arranged in a grid. Each detector is associatedrespectively with an indicator intersecting its associated detector.,e.g. detector (2501) is associated with indicator (2502). Each detectoris also associated respectively with an order entry item which may beselected by dwell. For example, detector (2501) is associated with ahamburger. Pointer (2512) (not drawn to scale) emits energy detectableby the detectors. Pointer (2512) may be housed in stationary housing(2514) (not drawn to scale).

Means for determining the difference between two data items or signalsand means for totaling two or more data items or signals may eachinclude a processing unit programmed to calculate this difference ortotal. Alternatively, the difference or total may be determined byelectronic, mechanical, optical, or other suitable means.

The two step procedure described earlier for operating conventionalmenu-driven data entry and order entry systems incorporating pointing atintended selections may be simplified to a single step in accordancewith the Dwell aspect of the invention. Referring now to FIG. 38, theoperator points pointer (2512) at an order entry item. The itembrightens responsive to the signals falling on the associated detector,indicating to the operator which order entry item he is dwelling on andhis dwell time on that order entry item. When his dwell time equals orexceeds the selection threshold, the order entry item is selected. Theoperation of such a system is intuitive and may be learned or relearnedby pointing and dwelling. No surface is required, unlike a standardmouse. Additionally, an operator seated in the driver's seat of avehicle is probably right-handed and may be making selections with hisleft hand, the hand closest to the window in a left-hand drive vehicle.Pointing and maintaining a pointer in a steady position requires lesscoordination than pointing and clicking.

D. Path Directness

The Path Directness aspect of the invention includes several aspects,hereinafter "subaspects", called Facilitated Dwell, Direction andIntersection, Direction, Appraisal and Drift Attenuation. The preferredembodiment and certain alternative embodiments of each of thesesubaspects will now be described.

According to the Facilitated Dwell subaspect of the invention, theduration of the dwell period required for selection ("selectionthreshold period") of a menu option associated with a selectable regionvaries with the directness of the cursor's path to that selectableregion. The preferred embodiment of the Facilitated Dwell subaspect ofthe invention will now be described in detail from a functionalperspective using an example illustrated in FIG. 39. In FIG. 39 ondisplay (2112) are defined corridors (2302), (2304), (2306), (2308),(2310), (2312) and (2314) lying between the previously selectedselectable region (2318) and, respectively, selectable regions (2342),(2344), (2346), (2348), (2350), (2352) and (2354). Each selectableregion is associated respectively with a selection threshold period.Each cursor location in a cursor path, for example cursor path (2322),may slightly decrease one or more the selection threshold periods,except for the selection threshold period associated with the previouslyselected selectable region, selectable region (2318) in this example.The effect of a cursor location depends, in the preferred embodiment, onwhether that cursor location intersects one of the corridors. If itdoes, as for example location (2324) intersects corridor (2308), thenthe selection threshold period associated with the selectable region towhich the intersected corridor leads, (2348) in this example, isdecreased, preferably to a limit of approximately 20% of the initialvalue of the selection threshold period so that some period ofintersection of the cursor and the intended selectable region is stillrequired for selection. A changed selection threshold period ispreferably indicated by a change in the brightness of the selectableregion associated with the changed selection threshold period. Thus,when the operator moves the cursor within a corridor, the selectableregion associated with the corridor brightens, indicating both thetarget selectable region the system believes the cursor is headed towardand the changed selection threshold period. When the operator moves thecursor outside a corridor the cursor had previously intersected, theselectable region associated with the previously intersected corridordarkens in accord with the dynamic dwell aspect of the invention,indicating both that the system no longer believes the cursor is headedtoward that selectable region and the changed selection thresholdperiod. Reducing the selection threshold period facilitates selection ofdwell-selectable regions without unduly increasing the likelihood oferroneous selections, since cursor locations within a corridor evidencethe operator's intention to select the selectable region associated withthe corridor.

Preferably, corridors are hidden from view, but they may be may be shownon the display or shown only at certain times or under certainconditions. Corridors may have fixed boundaries, depending on whichselectable region has been selected, or their boundaries may bedetermined when a starting location, for example, cursor location (2316)in FIG. 39, is known. Corridor shape, size, number and position aboutthe associated selectable region may vary, as illustrated by thealterative embodiments shown in FIGS. 40 and 41. Where corridorsoverlap, a cursor location intersecting two or more corridors may bedefined to be in a cursor path toward zero, one or more selectableregions associated with the intersected corridors.

The intersection of a cursor location and a corridor is but one means ofidentifying which one of a plurality of selectable regions is mostnearly along a cursor path. A cursor path may be indicated by anintersection of a cursor location and a predetermined region, e.g. acorridor, by a cursor location and a movement related signal from whichmay be derived a second location, or by two or more successive cursorlocations. As used herein, successive locations include a plurality oflocations distributed in time. Successive location may be, but need notbe, consecutive. Given a location and a movement related signal or twolocations, an intention to select a particular selectable region may beinferred, for example, by extrapolation, and the selection thresholdsassociated with either or both the intended or unintended selectableregions modified accordingly. As an example, assume successive cursorlocations are periodically stored in a ring buffer and the magnitude ofthe angle between two line segments, the first between the oldest cursorlocation in the ring buffer and a predetermined point in the selectableregion, and the second between the oldest cursor location in the ringbuffer and the current cursor location, is determined. The selectableregion associated with the smallest of these angles may be considered tobe the selectable region most nearly along the cursor path indicated bythe first line segment.

An alternative means of identifying which one of a plurality ofselectable region is most nearly along a cursor path is to determine theratio of the number cursor locations indicating a selectable region tothe total cursor locations in the cursor path.

FIG. 42 illustrates still another alternative means for identifyingwhich one of a plurality of selectable regions is most nearly along acursor path. FIG. 42 shows line (2410) from starting cursor location(2406) to ending cursor location (2408) just within selectable region(2404). Line (2410) is the most direct path between these two points. Inthis example, the actual path traveled by the cursor between these twopoints is path (2412). The ending cursor location (2408) is known beforethe selection threshold period associated with selectable region (2404)is modified. Identification may be made by measuring or approximatingthe area within region (2414) bounded by line (2410) and cursor path(2412). The smaller the area, the more direct the cursor path.Alternatively, identification may be made by storing a sampling amongsuccessive cursor locations along a cursor path and, in response to thecursor intersecting a selectable region, the selectable region mostnearly along the cursor path may be identified at one or more sampledpoints along the cursor path. The identification may be based upon aplurality of cursor locations or upon a single cursor location and amovement related signal. Alternatively, identification may be made bymeasuring and comparing the number of times a cursor path diverges froma predetermined path toward the intersected selectable region and/or thedegree of divergence of a cursor path from a predetermined path towardthe intersected selectable region.

Another apparatus in accord with the Path Directness aspect of theinvention is illustrated in FIG. 34. In this Figure, selectable regions(3101), (3103), (3105), (3107), (3109), (3111), (3115), (3117), (3119),(3121), (3125), (3127), (3129), (3131), (3133), (3135), (3139), (3141),(3143) and (3145) circumscribe area (3151). Area (3151) and optionallythe selectable regions include detectors for sensing radiant energyemitted from a pointer (2202) coupled to a body member of the operator.A computer coupled to the detectors determines which selectable regionis most nearly along the path indicated by the body member of theoperator. Responsive to the indicated path, the embodiment mayfacilitate the selection of one of the selectable regions by reducing aselection threshold, may select a selectable region upon intersection ofthe point indicated by the pointer (2202) and a selectable region, ormay select a particular selectable region in advance of intersection ofthe point indicated by the pointer (2202) and the particular selectableregion.

A move direction of a body member of an operator may be determined inany way that a cursor path may be determined, including sampling amongdata indicative of position of the body member. In determining movedirection of a body member, data indicative of body member positions mayserve the same function as cursor locations in indicating a path towarda selectable region. Position indicating means, used, for example, inindicating a position of the body member, includes each of the means forindicating that a selection event has occurred.

As an alternative to decreasing a selection threshold period, anembodiment may include a plurality of selection thresholds, eachassociated respectively with a selectable region. One or more of theselection thresholds may be increased when the direction of cursormovement does not indicate a path toward the associated selectableregion.

Means for indicating which one of the plurality of selectable regions ismost nearly along the cursor path includes each of the means forindicating an intersection of the location indicated by the movementrelated signal and a selectable region.

According to the Direction and Intersection subaspect of the invention,the selection threshold period is completely satisfied in response to acursor path to a particular selectable region, so that when the cursorintersects the particular selectable region, that region is selected.Preferably, the selection threshold period is completely satisfied inresponse to a measure of directness of a cursor path to a particularselectable region equalling or exceeding a predetermined measure ofdirectness. In circumstances where the measured directness is less thanthe predetermined measure, a dwell period is required for selection ofthe particular selectable region.

According to the Direction subaspect of the invention, a selectableregion is selected in response to a cursor path to that selectableregion, in advance of an intersection of the cursor and that selectableregion. Preferably, the selectable region is selected in response to ameasure of directness of a cursor path to a particular selectable regionequalling or exceeding a predetermined measure of directness. Incircumstances where the measured directness is less than thepredetermined measure, a dwell period is required for selection of theparticular selectable region.

According to the Appraisal subaspect of the invention, the directness ofa cursor path to a selectable region is measured. Preferably, the meansfor measuring the directness of a cursor path includes each of the meansfor identifying which one of a plurality of selectable regions is mostnearly along a cursor path. Thus, the particular means for identifyingwhich one of a plurality of selectable regions is most nearly along acursor path which best correlates with an operator's intended targetselectable region may be identified.

As earlier described, some NMD operators have relatively unimpaireddirectional control, despite having other movement disorders. TheFacilitated Dwell, Direction and Intersection, and Direction subaspectsof the Path Directness aspect of the invention utilize that capabilityfor computer access. Specifically, the ability of an operator to move acursor in a direct path toward a selectable region is used to facilitateselection of that selectable region. The selectable region is selectedmore quickly than in conventional systems utilizing selection by dwell,increasing operator productivity. In addition, when selectable regionsare located in accordance with the Perimeter Menu aspect of theinvention, a cursor path toward a selectable region is oftenunambiguous, since usually there is only one selectable region along acursor path, and a large rectangular area on the display is availablefor the output of an application program and is not obstructed by themenu. In certain embodiments in accordance with the Facilitated Dwell,Direction and Intersection, and Direction subaspects of the PathDirectness aspect of the invention, the operator may receive anindication of which selectable region the system believes the operatoris moving the cursor toward. The operator may adjust the cursor path inresponse to this feedback and thus move the pointer more accurately.Additionally, in embodiments in accordance with both the FacilitatedDwell subaspect of the Path Directness aspect of the invention and theDwell aspect of the invention, the operator may receive an indication ofthe dwell time required to select the selectable region most nearlyalong the cursor path as the required dwell time changes in response tothe cursor path.

The preferred embodiment of the Drift Attenuation subaspect of theinvention will now be described in detail from a functional perspectiveusing an example illustrated in FIG. 43. FIG. 43 depicts the upper rightcorner of a display (2112) having two selectable regions (0434) and(0430) thereon. Assuming, for purposes of this example, that a movementrelated signal indicates the path shown from point (0402) to point(0404) ("first segment") at a relatively high velocity and from point(0404) to point (0408) ("second segment") at a relatively low velocity.The path of the first segment is relatively direct, the path of thesecond segment, relatively meandering. During receipt of the movementrelated signal for the first segment, the cursor preferably tracks theexact path indicated by the movement related signal. During receipt ofthe movement related signal for the second segment, the movement of thecursor is attenuated, preferably so that the cursor does not leaveselectable region (0434) until it is selected.

Many NMD operators are unable to cleanly stop movement of a body member,resulting in a relatively slow or meandering path being indicated by themovement related signal. According to the Drift Attenuation subaspect ofthe invention, drift, that is, unintentional movement, indicated by themovement related signal following intentional movement is distinguishedfrom the intentional movement, and cursor movement responsive to thedrift is attenuated relative to cursor movement responsive tointentional movement. Thus unintentional movements of NMD operators arefiltered so that the cursor is displayed closer to the location intendedby the operator and drifting of the cursor into a nearby, butunintended, selectable region, is avoided, resulting in fewer errors dueto unintended selections.

The preferred embodiment of each subaspect of the Path Directness aspectof the invention will now be described in detail from an implementationperspective. Preferably, the Facilitated Dwell subaspect is implementedby modifications to the access program (1206) described in the detaileddescription of the Perimeter Menu aspect of the invention. Themodifications required are: (1) Additional state processing should beadded to ST₋₋ SELECTED to create a Windows® region in the shape of acorridor starting a predetermined distance from the selected selectableregion to each of the other selectable regions; (2) The event datastructure should be expanded to accommodate an event for a selectableregion along a cursor path; (3) The procedure CreateEvent should bechanged so that, before setting an event to EV₋₋ DECAY or EV₋₋ MOVEMENT,a test is made for the intersection of the cursor hotspot and eachcorridor, and, on finding such an intersection, a new event, EV₋₋CORRIDOR, is stored in the event data structure indicated by pEvent forthe selectable region associated with the intersected corridor; (4) Acolumn should be added to the state table so that each state which inthe prototype may receive either EV₋₋ MOVEMENT or EV₋₋ DECAY will onreceipt of EV₋₋ CORRIDOR drive that state machine to ST₋₋ CORRIDOR; (5)An additional variable, CorridorIncrement, preferably having an initialvalue one half the value of Increment, should be added the set of dataassociated with each state machine; and (6) A new state, ST₋₋ CORRIDOR,should be added to the procedure PocketFsm. The pseudo-code for stateprocessing in ST₋₋ CORRIDOR follows:

    ______________________________________                                        case ST.sub.-- CORRIDOR:                                                                   /* set state to previous state */                                             /* in preparation for the next */                                             /* state transition */                                           set pPocket->State to pPocket->PreviousState                                  incrernent pPocket->Color by                                                  pPocket->CorridorIncrement, but not above                                     pPocket->InitialColor plus 80% of the difference                              between pPocket->CrestTide and                                                pPocket->InitiaiColor                                                         if PPocket->Color was changed                                                  set pPocket->fPaint to TRUE                                                  set fInternalEvent to TRUE                                                    set Event to EV.sub.-- NULL                                                   break                                                                         ______________________________________                                    

An example of the selection of selectable region (2346) in accord withthe preferred embodiment of the Facilitated Dwell subaspect of theinvention will now be described with reference to FIG. 39, assuming thatthe prototype has been modified as described, that the operator has justselected selectable region (2318), that all state machines are in stateST₋₋ RESET, and that the operator moves the cursor hotspot from alocation inside selectable region (2318) to location (2316), a locationjust outside selectable region (2318). On receipt of the next WM₋₋ TIMERmessage, procedure PocketFsm is called with the indicator for the statemachine associated with selectable region (2318) and with event EV₋₋CROSS₋₋ OUT. Event EV₋₋ CROSS₋₋ OUT drives this state machine from itscurrent state, ST₋₋ RESET, to ST₋₋ EBB₋₋ TIDE. The pseudo-code for ST₋₋EBB₋₋ TIDE in procedure PocketFsm is a break statement, indicating thatno state specific action is taken at this time, other than thetransition to ST₋₋ EBB₋₋ TIDE. Control returns to Windows® (1204).

Shortly before or shortly after the state machine associated withselectable region (2318) receives EV₋₋ CROSS₋₋ OUT all other statemachines each receive EV₋₋ MOVEMENT and each makes the transition fromST₋₋ RESET to ST₋₋ EBB₋₋ TIDE.

Assume that the operator moves the cursor hotspot along path (2322)toward location (2332), a location outside all corridors, and then thenext WM₋₋ TIMER message is received. All state machines receive EV₋₋MOVEMENT and are driven from ST₋₋ EBB₋₋ TIDE to ST₋₋ DECAY to ST₋₋ EBB₋₋TIDE.

The state processing associated with these states has been described inthe description of the Perimeter Menu aspect of the invention.

Assuming that the operator moves the cursor hotspot along path (2322),the transition from ST₋₋ EBB₋₋ TIDE to ST₋₋ DECAY to ST₋₋ EBB₋₋ TIDE isrepeated every 54 milliseconds for each state machine until the operatormoves the cursor hotspot beyond location (2332). At this time theprocedure CreateEvent determines that the cursor hotspot intersectscorridor (2308) and that consequently EV₋₋ CORRIDOR should be sent tothe state machine associated with selectable region (2348). EV₋₋CORRIDOR drives this state machine from ST₋₋ EBB₋₋ TIDE to ST₋₋CORRIDOR. Stepping through the pseudo-code for ST₋₋ CORRIDOR, the ST₋₋CORRIDOR state sets pPocket→State to the value stored inpPocket→PreviousState, increments pPocket→Color bypPocket→CorridorIncrement, but not above pPocket→InitialColor plus 80%of the difference between pPocket→CrestTide and pPocket→InitialColor,sets pPocket→fPaint to TRUE, sets fInternalEvent to TRUE, and sets Eventto EV₋₋ NULL. Following the break statement, the while fInternalEventcondition is true and another state transition occurs. The new state isfound at aPocketFsm[ST₋₋ EBB₋₋ TIDE][EV₋₋ NULL], which equals ST₋₋ EBB₋₋TIDE. This path is unlike an ordinary state transition because thestarting state is set by ST₋₋ CORRIDOR. All transitions from ST₋₋CORRIDOR share this distinction. The state machine executes the code forthe new state, ST₋₋ EBB₋₋ TIDE, which is simply a break statement. Theprocedure PocketFsm determines that fpaint is TRUE, invalidates theclient rectangle and exits. As a result of invalidating the clientrectangle, Windows® (1204) sends a WM₋₋ PAINT message to the accessprogram (1206). On receipt of WM₋₋ PAINT, the access program (1204)checks the value of fpaint for each state machine, and if TRUE, setsfPaint to FALSE and redraws the visible subregion of the selectableregion associated with that state machine and any menu option locatedthereon. The color of the redrawn visible subregion is determined by thevalue of the Color variable for that state machine. The incrementedvalue of pPocket→Color results in a slight brightening of selectableregion (2348) and reduces the difference between pPocket→Color andpPocket→CrestTide, corresponding to the dwell period required to selectthe associated selectable region. After redrawing, control returns toWindows® (1204).

Shortly before or shortly after the state machine associated withselectable region (2348) receives EV₋₋ CORRIDOR, all other statemachines each receive EV₋₋ MOVEMENT and each makes the state transitionsfrom ST₋₋ EBB₋₋ TIDE to ST₋₋ DECAY to ST₋₋ EBB₋₋ TIDE.

This scenario is repeated at 54 millisecond intervals while the cursorhotspot travels along path (2322) to location (2326), a locationintersecting corridor (2308). Between this location (2326) and location(2328), a location intersecting corridor (2306), along path (2322), theprocedure CreateEvent determines that all state machines should receiveEV₋₋ MOVEMENT, driving each of them from their current state to ST₋₋DECAY and back to their current state. As described in the example inthe detailed description of the Perimeter Menu aspect of the invention,ST₋₋ DECAY state processing darkens the selectable region associatedwith the state machine, but not below a predetermined brightnessrepresented by the variable InitialColor. Thus selectable region (2348)darkens when the cursor hotspot no longer intersects corridor (2308).From location (2328) to location (2330), the procedure CreateEventdetermines that the state machine associated with selectable region(2346) should received EV₋₋ CORRIDOR and all other state machines EV₋₋MOVEMENT. Consequently, selectable region (2346) gradually brightens upto a ceiling represented by pPocket→InitialColor plus 80% of thedifference between pPocket→CrestTide and pPocket→InitialColor. Theduration of dwell time required for selection of selectable region(2346) is thus reduced to approximately 20% of the dwell period requiredwithout Facilitated Dwell.

Preferably, the Direction and Intersection subaspect of the invention isimplemented by making the changes to the prototype described for theFacilitated Dwell subaspect, except that, in incrementing pPocket→Colorin ST₋₋ CORRIDOR, the upper limit for pPocket→Color in ST₋₋ CORRIDORstate processing is pPocket→CrestTide minus pPocket→Increment. Assumingthese changes, a selectable region whose associated Color variable is atthis upper limit is selected during processing of the WM₋₋ TIMER messageimmediately following the intersection of the cursor hotspot and theselectable region.

Preferably, the Direction subaspect of the invention is implemented bymaking the changes to the prototype described for the Direction andIntersection subaspect of the invention, except that (1) the corridorsare narrow, as illustrated in FIG. 40; and (2) the pseudo-code fir stateprocessing in ST₋₋ CORRIDOR is as follows:

    ______________________________________                                        caseST.sub.-- CORRIDOR:                                                                    /* set state to previous state */                                             /* in preparation for the next */                                             /* state transition */                                           set pPocket->State to pPocket->PreviousState                                  increment pPocket->Color by                                                   pPocket->CorridorIncrement but not above                                      pPocket ->Ceiling                                                             if pPocket->Color was changed                                                  set pPocket->fPaint to TRUE                                                  set fInternalEvent to TRUE                                                    if pPocket->Color was changed from a value below                              pPocket->CrestTide to a value greater than or                                 equal to                                                                      pPocket->CrestTide                                                             set Event to EV.sub.-- STEP.sub.-- UP                                        else                                                                           set Event to EV.sub.-- NULL                                                  break                                                                         ______________________________________                                    

State ST₋₋ CORRIDOR may now generate the internal event EV₋₋ STEP₋₋ UP,as state ST₋₋ DWELL does in the detailed description of the PerimeterMenu aspect of the invention. A state machine in state ST₋₋ EBB₋₋ TIDEhaving a Color variable equal to or exceeding the CrestTide variable,will transition via PocketFsm[ST₋₋ EBB₋₋ TIDE][EV₋₋ STEP₋₋ UP] to ST₋₋SELECTED, and perform the ST₋₋ SELECTED state processing described inthe detailed description of the Perimeter Menu aspect of the invention.

Preferably, the Appraisal subaspect of the invention is implemented bymaking the changes to the prototype described for the Facilitated Dwellsubaspect, except that (1) the, corridors are visible on the display(2112), (2) one of the selectable regions is designated to be the targetselectable region and this is indicated to the operator, (3) cursorlocations are stored in memory (2106), and (4) following an intersectionof the cursor hotspot and the target or selection of a selectable regionother than the target, path directness is measured in accordance withthe stored cursor locations.

The preferred embodiment of the Drift Attenuation aspect of theinvention will now be described in detail from an implementationperspective. Preferably, the Drift Attenuation aspect is implemented bymodifications to the access program (1206) described in the detaileddescription of the Perimeter Menu aspect of the invention, modified asdescribed above in the description of the implementation of thepreferred embodiment of the Facilitated Dwell subaspect of the inventionand further modified as follows: (1) add two booleans to the data setassociated with each state machine, one called fDirectPath, the otherfAttenuateDrift, and initialize each of them in all state machines toFALSE; (2) in both ST₋₋ DECAY and ST₋₋ CORRIDOR, set bothpPocket→fDirectPath and pPocket→fAttenuateDrift to FALSE; (3) append toST₋₋ ENTRY state processing corresponding to the following pseudo-code:

    ______________________________________                                        if pPocket->Color equals or exceeds                                           (pPocket->InitialColor + (0.5 * (pPocket->CrestTide -                         pPocket->InitialColor))                                                        set pPocket->fDirectPath to TRUE                                             ______________________________________                                    

(4) append to ST₋₋ DWELL state processing corresponding to the followingpseudo-code:

    ______________________________________                                        if the present cursor position intersects the edge of the                     Windows ® cursor clipping rectangle and                                   pPocket->fDirectPath is TRUE                                                   set pPocket->fAttentuateDrift to TRUE                                        if pPocket->fAttentuateDrift is TRUE                                           store the difference in each of the x and y                                   coordinates ("delta") between consecutive cursor                              locations in a circular buffer accommodating the                              last ten deltas, overwriting the oldest delta with                            the newest delta                                                              if ten deltas have been accumulated                                            calculate the average acceleration indicated by                               the last ten deltas                                                           if the average acceleration is negative                                        set the cursor at the location one half the                                   distance between the current cursor location                                  and the previous cursor location                                              display the cursor at this new cursor                                         location                                                                   ______________________________________                                    

E. Intersection

The preferred embodiment of the Intersection aspect of the inventionwill now be described in detail from a functional perspective using anexample depicted in FIGS. 44, 45, 46. FIG. 44 shows the display (2112)of a general purpose computer system (2218 in FIG. 15) to which iscoupled a pointer (2202). Eight regions are delimited with respect tothe display (2112) and together circumscribe region (2650) on thedisplay (2112). Four of these regions, (2602), (2612), (2622), and(2632), are entirely on the display (2112). Each of the other fourregions respectively includes both a visible subregion ((2606), (2616),(2626), and (2636)) on the display (2112) and an invisible subregion((2604), (2614), (2624), and (2634)) adjacent and outside the display(2112). Assume, for example, that the pointer is indicating location(2656) on the display (2112) and that the operator moves the pointer sothat the location indicated by the pointer first intersects one of theregions at location (2652) in region (2612). Upon this intersection, thedisplay changes to that shown in FIG. 45. FIG. 45 depicts a selectableregion consisting of the union of invisible subregion (2714) and visiblesubregion (2716), hereinafter referred to as selectable region(2714/2716). Selectable region (2714/2712) is associated with menuoption "sumac". In the preferred embodiment, the operator may selectmenu option "sumac" by dwelling on any part of the selectable region forthe selection threshold period. Assuming, for example, that instead ofmoving the pointer from a position indicating location (2656) tolocation (2652), the operator instead moves the pointer so that thelocation indicated by the pointer first enters one of the regions atlocation (2654) in region (2614/2616). Upon this intersection, thedisplay changes to that shown in FIG. 46. FIG. 46 depicts a selectableregion consisting of the union of invisible subregion (2814) and visiblesubregion (2816), hereinafter referred to as selectable region(2814/2816). Selectable region (2814/2816) is associated with menuoption "vort<space>x". In the preferred embodiment, the operator mayselect menu option "vort<space>x" by dwelling on any part of theselectable region for the selection threshold period.

In the preferred embodiment, each of the other seven regions shown inFIG. 44 is associated with a selectable region and each selectableregion is associated with a menu option. The menu options shown in FIG.44, in addition to "sumac" and "vort<space>x" are "wizen", "backspace","words", "talk", ldhbfk" and "ypgqj,". The operator may select the menuoption associated with any one of the selectable regions by moving thelocation indicated by the pointer from circumscribed region (2650) intothe region associated with the selectable region and then dwelling onthe selectable region for the selection threshold period. Eachselectable region preferably includes all the area of its associatedregion.

An operator having impaired ability to maintain a body member in asteady position but who can control the point at which the locationindicated by a body member enters a region may, in accord with theIntersection aspect of the invention, use his relatively unimpairedmotor capability to selectively enlarge a selectable region or determinewhich of two or more selectable regions will occupy a predeterminedarea, thus making it easier for him to select. If the general purposecomputer system of the preferred embodiment is coupled to a speechsynthesizer and the menu options are letters or words, an operator withimpaired speech may select or spell words and speak them.

FIGS. 47 and 48 illustrate the upper right corner of a display inaccordance with an alternative embodiment of the Intersection aspect ofthe invention. FIG. 47 depicts; the upper right corner of a display(2112) of a computer system (2116) having thereon two regions, (1404)and (1402), each associated respectively with a selectable region, eachselectable region associated respectively with menu options "bdfhkl" and"<space>cmnrst". The selectable region associated with region (1402)includes all of region (1402). The selectable region associated withregion (1404) includes all of region (1404) plus area (1406) betweenregion (1404) and the right edge of the display (2112). Assuming theoperator uses a pointer to indicate a location on the display and thatthe location to first intersect the union of regions (1402) and (1404)intersects region (1402), the display remains as shown in FIG. 47.Dwelling in area (1406) operates to select the menu option"<space>cmnrst". If instead the location to first intersect the union ofregions (1402) and (1404) intersects region (1404), the display changesto that shown in FIG. 48. Dwelling in area (1406) then operates toselect the menu option "bdfhkl".

The preferred embodiment of the Intersection aspect will now bedescribed in detail from an implementation perspective. The Intersectionaspect is preferably implemented by modifications to the access program(1206) described above in the detailed description of the Perimeter Menuaspect of the invention. The modifications are: (1) add a row to thestate table aPocketFsm at row index 18 for a new state, ST₋₋ EXIT andinitialize the value of aPocketFsm[ST₋₋ EXIT][EV₋₋ NULL] to 4, the valueof ST₋₋ LOW₋₋ TIDE; (2) at initialization time, change aPocketFsm[ST₋₋LOW₋₋ TIDE][EV₋₋ CROSS₋₋ OUT] to 18, the value of ST₋₋ EXIT; (3) atinitialization time for each state machine, (a) if the visible subregionassociated with the state machine does not abut the edge of the display,set fInterior to TRUE, otherwise set fInterior to FALSE; (b) initializeiAdjacentPocket to the index of the state machine associated with theadjacent region. For example, assuming the index of the state machineassociated with region (2614/2616) in FIG. 44 is 2 and the index of thestate machine associated with region (2612) in FIG. 44 is 3,iAdjacentPocket in the data set associated with the state machineassociated with region (2614/2616) is initialized to 3 andiAdjacentPocket in the data set associated with the state machineassociated with region (2612) is initialized to 2; (c) initialize thesecond array of points to define the boundaries of the region associatedwith the selectable region associated with each state machine, forexample, region (2612), and initialize the first array of points todefine the boundaries of the visible subregion of the selectable regionassociated with the state machine, for example, visible subregion(2712); (4) remove creation of Windows® regions corresponding to visiblesubregions from the access program (1206) initialization; (5) append toST₋₋ ENTRY state processing corresponding to the following pseudo-code:

    ______________________________________                                        if pPocket->fInterior equals TRUE                                              if pPocket->hRegion is not NULL                                                delete the Windows ® region having the handle                             pPocket->hRegion                                                             create a Windows ® region having the boundaries                           defined by the second array of points associated                              with this state machine and set pPocket->hRegion to                           the handle to this region                                                     set pPocket->fPaint to TRUE                                                  send EV.sub.-- RESET to all state machines except this one and                the state machine having the index                                            pPocket->iAdjacentPocket                                                      set pAdjacentPocket to point to the data set associated                       with the state machine having the index                                       pPocket->iAdjacentPocket                                                      delete the Windows ® region having the handle                             pAdjacentPocket->hRegion                                                      set pAdjacentPocket->hRegion to NULL                                          set pAdjacentPocket->fPaint to TRUE                                           ______________________________________                                    

(4) append to ST₋₋ RESET state processing corresponding to the followingpseudo-code:

    ______________________________________                                        if pPocket->hRegion is not NULL                                                delete the Windows ® region having the handle                             pPocket- >hRegion                                                            create a Windows ® region having the boundaries defined by                the second array of points associated with this state                         machine and set pPocket->hRegion to the handle to this                        region                                                                        ______________________________________                                    

An example of the selection of selectable region in accord with theIntersection aspect of the invention will now be described withreference to FIGS. 44 and 45. Following initialization, assume that theoperator moves the pointer (2202) from a position indicating location(2656) toward location (2652) in FIG. 44. At the next expiration of thecursor polling timer, the procedure CreateEvent sends an EV₋₋ MOVEMENTevent to the state machine associated with the selectable regionassociated with the menu option "sumac". The event EV₋₋ MOVEMENT drivesthis state machine from its current state, ST₋₋ RESET, to ST₋₋ EBB₋₋TIDE. The pseudo-code for ST₋₋ EBB₋₋ TIDE in procedure PocketFsm is abreak statement, indicating that no state specific action is taken atthis time, other than the transition to ST₋₋ EBB₋₋ TIDE. Control returnsto Windows® (1204). At the expiration of the cursor polling timer afterthe location indicated by the pointer reaches location (2652), theprocedure CreateEvent sends event EV₋₋ DWELL to the state machineassociated with the selectable region associated with the menu option"sumac". EV₋₋ DWELL drives this state machine from ST₋₋ EBB₋₋ TIDE toST₋₋ ENTRY. Following the pseudo-code for ST₋₋ ENTRY state processingshown above and in the Procedure PocketFsm, fInternalEvent is set toTRUE, Event is set to EV₋₋ NULL, the Windows® region corresponding toregion (2612), ie. the Windows® region having the handlepPocket→hRegion, is deleted and a new Windows® region having theboundaries indicated in the first array of points, ie. visible subregion(2712) in FIG. 45, is created and the handle stored in pPocket→hRegion.pPocket→fPaint is set to TRUE, EV₋₋ RESET is sent to state machinesassociated with regions other than (2612) and (2614/2616). Data in thedata set associated with the adjacent region (2614/2616) is nowmodified. The Windows® region corresponding to region (2614/2616) isdeleted, hRegion for the state machine associated with that region isset to NULL and fpaint associated with that state machine is set toTRUE. The event EV₋₋ NULL drives the state machine associated withselectable region (2714/2712) to ST₋₋ LOW₋₋ TIDE. The pseudo-code forST₋₋ LOW₋₋ TIDE is only a break statement, so there is no state specificaction for ST₋₋ LOW₋₋ TIDE other than entry into this state.pPocket-fPaint is TRUE so the client rectangle is invalidated, resultingin the redrawing of Windows® regions associated with state machineshaving fpaint equal to TRUE so that the display (2112) appears as shownin FIG. 45. Control returns to Windows® (1204).

Dwelling at any location within selectable region (2714/2712) now causethe procedure CreateEvent to send EV₋₋ DWELL to the state machineassociated with this selectable region. Selection of this selectableregion proceeds as described above in the detailed description of thePerimeter Menu aspect of the invention.

Assuming the operator moves the location indicated by the pointer out ofselectable region (2714/2712) prior to selection, the procedureCreateEvent sends the event EV₋₋ CROSS₋₋ OUT to the state machineassociate with that selectable region, driving it to ST₋₋ EXIT. Thestate processing for ST₋₋ EXIT causes the display (2112) to change tothat shown in FIG. 44.

F. Alignment

The embodiment of the Alignment aspect of the invention as implementedin the prototype will now be described in detail from a functional andimplementation perspective using an example depicted in FIGS. 49, 50,and 51. Each of these Figures depicts the upper right corner of adisplay (2112) having two visible subregions on the display. In theseFigures, no subregions outside the display (2112) are shown. Alignmentis achieved in several steps and requires operator interaction with theapparatus. For purposes of this example, assume that FIG. 49 depicts theupper right corner of the display (2112), that an operator, fitted witha head pointer, desires to keep the cursor on the display directly inhis line of sight and that the location indicated by the pointer ispresently 15 degrees to the right of the location of the arrow cursor(1802). The operator now dwells on selectable region (1834) for anpredetermined period ("the lock threshold") which preferably issignificantly greater than the selection threshold period. The apparatusresponds to this extended dwell period by changing the display to thatshown in FIG. 49. The arrow cursor is removed from the display and thelock icon (1902) is displayed in a predetermined location of theintersected selectable region ("lockspot") on the display. The lock iconremains on the lockspot for a predetermined period ("the lock period");it does not move responsive to the operator's head movement. While thelock icon is displayed, the operator turns his head, bringing his lineof sight into alignment with the lockspot. At the expiration of the lockperiod, the apparatus changes the display to that shown in FIG. 51. Thelock icon is erased and the arrow cursor appears in the lockspot, whichis where the operator is now looking. The arrow cursor moves in responseto the operator's head movement.

An operator who loses alignment between location indicated by hispointer and the cursor may thus initiate an alignment sequence, andthen, by moving his head or other body member when the prototypeindicates he should do so by displaying the lock icon, regain alignment.That the lock icon is displayed indicates to the operator that he canalign his head or other body member. The position of the lock iconindicates to the operator the location on the display with which heshould align his head or other body member. This is the location wherethe arrow cursor will appear at the expiration of the lock period.

The Alignment aspect will now be described in detail from animplementation perspective as implemented in the prototype. TheAlignment aspect of the invention is implemented an integral part of thestate machine described in the detailed description of the PerimeterMenu aspect of the invention, using the same state machines andinitialization, with the exception that aPocketFsm[ST₋₋ CREST₋₋TIDE][EV₋₋ CEILING] is changed from 9, the value of ST₋₋ DWELL, to 14,the value of ST₋₋ BEGIN₋₋ LOCK. The operator initiates the alignmentprocess by moving the cursor so that the cursor hotspot intersects aselectable region. As an example, FIG. 49 shows the arrow cursor (1802)intersecting selectable region (1834). The operator then dwells on theselectable region for the lock threshold, preferably at least one secondgreater than the selection threshold period. This operator action causesthe state machine associated with the intersected selectable region toreach state ST₋₋ CREST₋₋ TIDE. On receipt of the first WM₋₋ TIMERmessage following entry into state ST₋₋ CREST₋₋ TIDE, the procedureCreateEvent creates event EV₋₋ CEILING, which drives the state machinefrom state ST₋₋ CREST₋₋ TIDE to state ST₋₋ BEGIN₋₋ LOCK. The stateprocessing within state ST₋₋ BEGIN₋₋ LOCK beeps, sets the system cursorlocation to the selectable region's lockspot, sets the system cursor tonull erasing the arrow cursor, displays a the lock icon (1902 in FIG.50) on the selectable region's lockspot and initializes the globalvariable iLockCursor to the number of expirations of the cursor pollingtimer corresponding to the period of time the cursor will be locked (the"lock period") configured by the operator, preferably two seconds, andcontrol is returned to Windows® (1204). Following the next expiration ofthe cursor polling timer, the procedure CreateEvent may generate anevent EV₋₋ DECAY, EV₋₋ DWELL, EV₋₋ CROSS₋₋ OUT, EV₋₋ MOVEMENT or EV₋₋CEILING. Each of these events drives the state machine from state ST₋₋BEGIN₋₋ LOCK to state ST₋₋ LOCK. Within state ST LOCK the system cursoris moved to the selectable region's lockspot and iLockCursor isdecremented. Then it is determined whether iLockCursor equals zero. Ifnot, control returns to Windows® (1204) for another iteration throughST₋₋ LOCK. During the lock period, the system cursor is moved to thelockspot upon every expiration of the cursor polling timer, thusinhibiting movement of the cursor from the lockspot so that, during thelock period, the operator may move the location indicated by the pointerwhile the lock icon remains on or very close to the selectable region'slockspot. When, eventually, iLockCursor is decremented to zero, theglobal Event is set to EV₋₋ NULL and fInternalEvent is set to TRUE sothat another state transition occurs immediately. This transition drivesthe state machine from state ST₋₋ LOCK to state ST₋₋ END₋₋ LOCK. Withinstate ST₋₋ END₋₋ LOCK, the lock icon is erased, Windows® (1204) isdirected to display the arrow cursor (2002) at the lockspot, as shown inFIG. 51, the fpaint flag associated with the intersected selectableregion is set to TRUE so that the selectable region will be drawn,restoring the background behind the erased lock icon. Assuming thecursor hotspot remains on the selectable region for another 54milliseconds, CreateEvent generates EV₋₋ DWELL, which drives the statemachine to ST₋₋ DISCARD. Referring to the pseudo-code in PocketFsm, ST₋₋DISCARD state processing sets State to PreviousState, returning thestate machine to ST₋₋ END₋₋ LOCK and in effect, discarding the lastevent. Following the expiration of the cursor polling timer after theoperator moves the cursor out of the selectable region it presentlyintersects, the procedure CreateEvent generates an event EV₋₋ CROSS₋₋OUT which drives the state machine from state ST₋₋ END₋₋ LOCK to stateST₋₋ SELECT₋₋ AND₋₋ OUT. The access program (1206) performs the staleprocessing for state ST₋₋ SELECT₋₋ AND₋₋ OUT and subsequent states aspreviously described in the description of the Perimeter Menu aspect ofthe invention.

In the prototype, the operator may initiate alignment by depressing anycharacter key on the keyboard. On receipt of a WM₋₋ CHAR message fromWindows® (1204), the access program (1206) removes the arrow cursor fromthe display and displays the lock icon at a predetermined location onthe display. In the prototype, the predetermined location is the centerof the display (2112). After a predetermined period, two seconds in theprototype, the lock cursor icon is erased and the arrow cursor displayedat the predetermined location.

In the prototype, the cursor is automatically centered if the cursorhotspot does not move for two minutes. Lack of movement of the cursorhotspot is detected in the procedure CreateEvent, which generates theevent EV₋₋ IDLE₋₋ TIMEOUT for all state machines. The state processingof each state machine on receipt of EV₋₋ IDLE₋₋ TIMEOUT depends upon itscurrent state. State machines in states ST₋₋ INITIAL and ST₋₋ RESET stayin those states. State machines in all other states in which an externalevent can be received are driven to ST₋₋ IDLE. Referring to thepseudo-code in PocketFsm, ST₋₋ IDLE state processing moves the cursor tothe center of the display, sets fInternalEvent to TRUE and sets Event toEV₋₋ NULL. PocketFsm[ST₋₋ IDLE][NULL] equals 3, the value of ST₋₋ RESET.The access program (1206) performs the state processing for the ST₋₋RESET and subsequent states as described in the description of thePerimeter aspect of the invention.

G. Length Order

The preferred embodiment of the Length Order aspect of the inventionwill now be described in detail from a functional and implementationperspective using an example depicted in FIG. 52 and described in thedetailed description of the Location Indication aspect of the invention.Assume, for purposes of this example, that the operator has previouslyselected the letter "s". In the preferred embodiment, the string "s" ispassed from the access program (1206 in FIG. 16) via the Windows®Dynamic Data Exchange ("DDE") interface to a database program (1210 inFIG. 16), preferably the FoxPro for Windows® version 2.6 program,available from Microsoft Corporation, Redmond, Wash., USA. The FoxProprogram looks up the record having the index "s" in a database composedof 26 letters, one for each letter of the alphabet. Each record includesone field for a letter of the alphabet and 12 fields containing the 12most frequently used English words beginning with that letter. The 12words in the record are ordered primarily by length, determined by thenumber of letters in each word, and secondarily by alphabetic order. TheFoxPro program returns these 12 words to the access program (1206) andthese are displayed on the display (2112) as named menu options. Thetwelve named menu options, ordered as described and depicted in FIG. 52,are "so", "say", "she", "said", "show", "some", "such", "state","school", "social", "service", and "student". Preferably, only the rootform of inflected forms of words which can be created through availablesuffixes ("root word") may be displayed as named menu options, so thatthe limited number of available menu options in combination with theapparatus' capability to add suffixes offers a large number of inflectedforms.

Words which may be named menu options in the same menu may be ordered byany suitable method. Preferably ordering is done by an ordering programoperating on a corpus of text or speech including text or speechproduced by individuals whose age, sex, geographic location anddisability are the same as or similar to that of the operator. Theordering program determines the frequency of use of root words in thecorpus, selects the twelve most common root words, beginning with everypossible combination of one, two and three letters, and stores them inthree FoxPro databases for one, two and three letter word beginningsrespectively, the words in each record ordered as described above.Ordering the words prior to a request minimizes the delay between theoperator's selection of a letter or letters and the display of the namedmenu options. Preferably, the ordering program also creates a databaseof records for root words beginning with four or more letters. Eachrecord includes the words and its frequency of use in the corpus. Whenthe operator selects four or more letters consecutively, the accessprogram (1206) requests via DDE that the FoxPro program (1210) look upwords starting with four or more letters in the database for wordsbeginning with four or more letters, select the 12 most frequently usedwords matching the selected letters, order them as described above, andreturn them to the access program (1206).

An operator searching named menu options for a desired menu option maystart his search in the area on the display most likely to contain thedesired menu option. Upon comparison of the length of the desired menuoption to a displayed menu option, the operator may determine whether tocontinue his search from the displayed menu option toward the front ofthe list or toward the rear of the list, or to jump to another displayedmenu option in the list Further, he may make this determination morequickly than if the displayed menu options were sorted conventionally.The reduced menu option search time increases the operator'sproductivity with respect to conventional menu interfaces.

H. Location Indication

The preferred embodiment of the Location Indication aspect of theinvention will now be described in detail from a functional perspectiveusing an example depicted in FIG. 52. FIG. 52 shows the display (2112)of a general purpose computer system (2218 in FIG. 15) having thereoneight selectable regions, each associated respectively with a menuoption. Each of the eight selectable regions consists of the union of avisible subregion on the display (2112) and an invisible subregion (notshown) located outside the display (2112) and adjacent the visiblesubregion. Selectable region (4301) is associated with menu option "so",selectable region (4303) with menu option "say", selectable region(4305) with menu option "she", selectable region (4307) with menu option"suffixes", selectable region (4309) with menu option "words",selectable region (4311) with menu option "some", selectable region(4313) with menu option "show" and selectable region (4315) with menuoption "said". Also shown on the display (2112) are 12 indicatingregions, each associated respectively with a menu option. Indicatingregion (4351) is associated with menu option "so", indicating region(4353) with menu option "say", indicating region (4355) with menu option"she", indicating region (4361) with menu option "some", indicatingregion (4363) with menu option "show", indicating region (4365) withmenu option "said", indicating region (4371) is associated with menuoption "such", indicating region (4373) with menu option "state",indicating region (4375) with menu option "school", indicating region(4381) with menu option "student", indicating region (4383) with menuoption "service" and indicating region (4385) with menu option "social".Each indicating region is located on the display (2112) in the region(4350) circumscribed by the selectable regions. In accord with theLocation Indication aspect of the invention, the location of each of theindicating regions (4351), (4352), (4355), (4361), (4363) and (4365)indicates the location of each of the selectable regions associated withthe menu option associated with the indicating region. These selectableregions, respectively, are (4301), (4303), (4305), (4311), (4313) and(4315).

In the preferred embodiment, selection of menu option "words" causesselectable region (4301) to be associated with menu option "such"instead of menu option "so", selectable region (4303) to be associatedwith menu option "state" instead of menu option "say", selectable region(4305) to be associated with menu option "school" instead of menu option"she", selectable region (4311) to be associated with menu option"student" instead of menu option "some", selectable region (4313) to beassociated with menu option "service" instead of menu option "show", andselectable region (4315) to be associated with menu option "social"instead of menu option "said". In the preferred embodiment, followingthe selection of menu option "words", the menu option newly associatedwith each selectable region is displayed on that selectable region (notshown). Following the selection of menu option "words", indicatingregions (4371), (4373), (4375), (4381), (4383) and (4385) each indicatethe location of each of the selectable regions associated with the menuoption associated with the indicating region.

The indicating regions and the menu options displayed thereon in FIG. 52are disproportionately large relative to the rest of FIG. 52. They areapproximately 1.5 times their proportionate size. They are representedas shown in FIG. 52 in compliance with Patent Cooperation Treaty Rulesrequiring a minimum size for letters in figures.

In the preferred embodiment of the Perimeter Menu aspect of theinvention, all the menu options displayed on the selectable regionscannot be seen in a glance by many operators. However, when thedisplayed menu options are displayed on the indicating regions, forexample, as shown in FIG. 52, the displayed menu options can be seen ina glance, facilitating searching of the menu by the operator for hisintended menu option. Since the location of each selectable region isindicated by the location of the associated indicating region, theoperator may point to the intended selectable region without searchingit out or pausing to verify that the menu option associated with aselectable region is the menu option he desires. The frequent operatorof such a menu interface may habituate the process of selecting anintended menu option so he can focus his attention on another task whileselecting the option, for example, planning his next interaction withthe menu interface.

FIG. 53 depicts a display and structures in accord with an anotherembodiment of the Location Indication aspect of the invention.Selectable regions (4604), (4608), (4612), (4616), (4620), (4624),(4628) and (4632) are located adjacent the display (2112) and associatedrespectively with menu options "so", "say", "she", "suffixes", "said","show", "some" and "words". Each selectable region is unbounded on theside furthest from and parallel to the edge of the display. Indicatingregions (4351), (4353), (4355), (4357), (4367), (4361), (4363) and(4365) are associated respectively with menu options "so", "say", "she","suffixes", "said", "show", "some" and "words". In accord with theLocation Indication aspect of the invention, the location of anindicating region indicates the location of the selectable regionassociated with the menu option associated with the indicating region.

The large size of the selectable regions outside the display, forexample, as shown in FIG. 53, facilitate selection by individuals withimpaired fine motor control while the indicating regions indicate thelocation of each associated selectable region.

Yet another embodiment in accord with the Location Indication aspect ofthe invention is shown in FIGS. 54 and 55. FIG. 54 depicts a display(2112) having thereon eight selectable regions, each associatedrespectively with a menu option. As shown in the Figure, selectableregion (3911) is associated with menu option "aeiou", selectable region(3910) with menu option "gqjyvwxz", selectable region (3908) with menuoption "setup", selectable region (3907) with menu option undo,selectable region (3905) with menu option "control", selectable region(3904) with menu option "words", selectable region (3902) with menuoption "nsrm<space>tcp" and selectable region (3901) with menu option"bdfhkl". Menu option "aeiou" is associated with a submenu whichincludes submenu options "a", "e", "i", "o" and "u", displayed onindicating regions (3930), (3928), (3926), (3924) and (3922)respectively. The background pattern of indicating region (3930) matchesthe background pattern of selectable region (3908), indicating thatsubmenu option "a" will be associated with selectable region (3908)following selection of menu option "aeiou". Similarly, the backgroundpattern of indicating region (3928) matches the background pattern ofselectable region (3910), indicating that submenu option "e" will beassociated with selectable region (3910) following selection of menuoption "aeiou". Alternatively, the indication may be made by the size,shape, hue, brightness, contrast, dithering, fill, blinking, hatching orpattern of the indicating region or any part thereof, including eitherof the foreground and background of the indicating region.

When the operator selects selectable region (3911), the display (2112)changes to that shown in FIG. 55. Now, selectable region (3910) isassociated with submenu option "e", selectable region (3908) withsubmenu option "a", selectable region (3904) with submenu option "u",selectable region (3902) with submenu option "o" and selectable region(3901) with submenu option "i". In accord with the Location Indicationaspect of the invention, the appearance of an indicating regionindicates the location of the selectable regions associated with thesubmenu option associated with the indicating region.

The embodiment shown in FIGS. 54 and 55 illustrates how the LocationIndication aspect of the invention speeds selection of a submenu optionof a menu hierarchy. The operator, by observing the background patternof the submenu option within the displayed menu option, may determinewhich selectable region he should next dwell on. The operator may makethis determination prior to selection of the menu option and need notwait for the submenu options to be displayed on their associatedselectable regions. In accordance with the Location Indication aspect ofthe invention, the operator may select or spell out words more quicklythan with conventional automated menu hierarchy. Assuming the display(2112) is part of computer system, these words may be input to anapplication program, and, if the computer system is coupled to a speechsynthesizer, these words may be spoken.

FIG. 56 depicts another apparatus in accord with the Location Indicationaspect of the invention. An example of the operation of this apparatuswill now be described. FIG. 56 depicts display area (4770) adjacent towhich are located selectable regions (4701), (4703), (4705), (4707),(4709), (4711), (4713) and (4715). Not shown in the Figure are threesets of eight menu options. At different times during the operation ofthe apparatus shown in FIG. 56, for example at one second intervals, theplurality of selectable regions is associated with a different set ofeight menu options such that, for a one second period, each selectableregion is associated respectively with one menu option of the associatedset of menu options. On the display area are 24 indicators, eachassociated respectively with one of the menu options. Each indicatorindicates when a selectable region is associated with the menu optionassociated with the indicator. Thus, each of the indicators (4721),(4723), (4725), (4727), (4729), (4731), (4733) and (4735) is associatedrespectively with a menu option which may be in turn associatedrespectively, for example, during a first one second period, with one ofthe selectable regions. The operator may select a desired one of the 24menu options by selecting the associated selectable region during theperiod when the desired menu option is associated with the associatedselectable region.

Alternatively, the apparatus shown in FIG. 56 may require two successiveselection events, the first selection event to select a set of eightmenu options or to select a set of three menu options, the set beingassociated with one selectable region, and the second selection event toselect one of the selected set. For example, the first selection eventmay select the set of menu options associated with indicators (4773),(4753) and (4733), and the second selection event may select one menuoption from this set. Still another apparatus in accord with theLocation Indication aspect of the invention is shown in FIG. 57. FIG. 57depicts display area (4770) outside of which are located selectableregions (5001), (5003), (5005), (5007), (5009), (5011), (5013), (5015),(5021), (5023), (5025), (5027), (5029), (5031), (5033) and (5035). Thelocation of each selectable region is indicated by an indicating regionon the display area, (5041), (5043), (5045), (5047), (5049), (5051),(5053), (5055), (5061), (5063), (5065), (5067), (5069), (5071), (5073)and (5075), respectively.

The Location Indication aspect of the invention will now be describedfrom an implementation perspective with reference to FIG. 52.Preferably, the Location Indication aspect is implemented bymodifications to the access program (1206) described in the detaileddescription of the Perimeter Menu aspect of the invention. Themodifications required for the Location Indication aspect of theinvention are: (1) at initialization time: (a) create twelve child editwindows, each corresponding to one of the indicating regions shown inFIG. 52, each of the class "edit", each having the style (WS₋₋ CHILD |WS₋₋ BORDER | WS₋₋ DISABLED | ES₋₋ MULTILINE | AlignmentStyle) where "|"represents a logical OR operation and where AlignmentStyle equals ES₋₋RIGHT for child windows having right justified text and ES₋₋ LEFT forchild windows having left justified text, as shown in FIG. 52; (b) movethe child edit windows so they are located in or near the center of thedisplay; (c) display the child edit windows; and (d) initialize an arraywhich maps each child edit window, for example, by an index between 0and 11 inclusive, to an element of the aLabel array; and (2) modify thecode implementing ST₋₋ SELECTED state processing, so that after changingthe menu options associated with various selectable regions (by changingthe pLabel element of the data set associated with each of the statemachines to point to the menu option in the aLabel array to beassociated with that state machine), the array mapping each child windowto an element of the aLabel array is used to access the text of the menuoption and the text of each child edit window is set accordingly. In thepreferred embodiment, the six menu options displayed in indicatingregions (4371), (4374), (4375), (4381), (4383) and (4385) are present inthe aLabel array when FIG. 52 is shown, but are not associated withselectable regions until the operator selects menu option "words".

I. Sound Match

The preferred embodiment of the Sound Match aspect of the invention willnow be described in detail from a functional perspective using anexample depicted in FIG. 58. FIG. 58 shows a display (2112) of acomputer system (2116) having thereon six regions or sound indicators,(5801), (5803), (5805), (5807), (5809), and (5811), each associated witha menu option, "ypgqj,", "ldhbfk", "wizen", "words", "vort x", and"sumac", respectively. Each menu option is displayed on its associatedsound indicator. Each sound indicator has a distinct hue. For example,sound indicator (5801) may be green, (5803) white, (5805) blue, (5807)red, (5809) orange and (5811) grey. Each of these sound indicatorsindicates a sound the operator is able to consistently produce, forexample, the vowel sound e as it sounds in green, i as it sounds inwhite, u as it sounds in blue, e as it sounds in red, o as it sounds inorange and e it sounds in grey. Assuming the operator says o as itsounds in orange, "vort x", the menu option associated with the orangesound indicator (5809) is selected and the display is changed to thatshown in FIG. 59. In the preferred embodiment, submenu options space,"x", "r", "t", "v" and "o" are now associated with each of the soundindicators (5901), (5903), (5905), (5907), (5909), and (5911)respectively. Assuming the operator says e as in green, submenu optionspace is selected and a space is input to the application program (6107)whose output appears in region (5850).

FIG. 60 illustrates a display and structures in accordance with analternative embodiment of the Sound Match aspect of the invention. FIG.60 depicts a display (2112) of a computer system (2116). Adjacent thedisplay (2112) is a static display (6252) having thereon eight soundindicators (6204), (6208), (6212), (6216), (6220), (6224), (6228), and(6232). Each of the eight sound indicators is a symbol in a phoneticcharacter set graphically representing a sound. Alternatively, the soundindicators may include a picture of an object, for example, a tree, ahouse, a boy, or a map of a country, or may include a shape, forexample, a square, a circle, or a triangle, or may include a hueindicator, a pitch indicator, a volume indicator, a sound durationindicator, a change in pitch indicator, or a change in volume indicator.Eight menu options, "vort<space>x", "sumac", "wizen", undo, "words","talk", "ldhbfk" and "ypgqj,", are displayed on the display (2112), eachadjacent and associated respectively with one of the sound indicators,the eight menu options together circumscribing region (6250) on thedisplay. The operator may select any one of the menu options byproducing the sound indicated by the sound indicator associated with themenu option. In response to a menu selection, submenu options of theselected menu option may be displayed and associated respectively with asound indicator.

An operator with impaired speech but who is able to consistently producea relatively small number of sounds distinguishable by conventionalspeech recognition means may, in accord with the Sound Match aspect ofthe invention, select from among the small number of menu options byusing the sounds he can produce. Assuming the display (2112) is part ofcomputer system, the menu option may represent inputs to an applicationprogram, and, if the computer system is coupled to a speech synthesizer,the menu option may represent words to be spoken.

FIG. 61 illustrates a display and structures in accordance with analternative embodiment of the Sound Match aspect of the invention. FIG.61 depicts a display (2112) of a computer system (2116) having thereonsix square sound indicators, (6401), (6403), (6405), (6411), (6413), and(6415) arranged in two columns of three sound indicators. Each soundindicator abuts a sound indicator in the other column, the soundindicator located above it, if any, and the sound indicator locatedbelow it, if any. Assume that the operator is entering Chinese, that heuses a keyboard to enter a phonetic unit and intonation according to thePin Yin coding method for the Chinese language, and that he enters thedistinct sound "fu", each of the sound indicators shown in FIG. 61 isassociated respectively with a menu option, each of the plurality ofmenu options having a common characteristic. In this example, the menuoptions are homophones and the common characteristic is a phonetic unitand intonation but may be a phonetic unit alone, an intonation alone, astroke used to draw an ideograph, a number of horizontal strokes, anumber of vertical strokes, a number of total strokes, a stroke order, aradical, a part of speech, an ideograph, a kana, a diacritic, aclassification of a part of an ideograph or other characteristic of aclass of ideographs. Each of the sound indicators has a distinct hue.The operator may select any one of the displayed menu options, accordingto the Sound Match aspect of the invention, by speaking the soundassociated with the sound indicator associated with the desired menuoption. For example, assume the operator says, "blue" or a translationthereof, preferably a Chinese translation in this example. The ideographlocated on the blue sound indicator is selected. In this example, thisideograph is input to a word processing program and appears on thedisplay (2112).

The Sound Match aspect of the invention thus allows an operator toselect from an option from a menu, using speech recognition means,whether or not the menu options are homophones. The operator does notneed to use his hands to make this selection and so may keep both hishands on the home row of the keyboard, in preparation for entering thenext common characteristic, or, if specifying the common characteristicby voice, may select a menu option without interrupting the manualactivity he's engaged in.

The preferred embodiment of the Sound Match aspect of the invention willnow be described in detail from an implementation perspective, beginningwith the hardware and software operating environment which will now bedescribed with reference to FIG. 62 which depicts a speech recognitionsystem (6001) including a computer system (2116) and keyboard (2210), asearlier described, a sound board (6007) and a microphone (6009). Theconventional computer system (2116) preferably includes an 80486 CPUrunning at 33 MHz or faster, and is provided with Dragon Dictate PowerEdition software, available from Dragon Systems, Inc., Newton, Mass.,USA. Preferably, the sound board is the Audio Capture Playback Adaptorand the microphone is the Shur Headset microphone, both available fromDragon Systems, Inc. FIG. 63 depicts the software environment of thepreferred embodiment, which includes the earlier described Windows®version 3.1 operating system (1204), an optional Windows® applicationprogram (6107), the Windows® Dragon software driver (6101 ) included inthe Dragon Dictate Power Edition software, the Dragon Application (6103)included in the Dragon Dictate Power Edition software, and a speechrecognition access program (6105). The Dragon Application (6103) isconfigured to match a sample of each of six sounds distinguishable bythe Dragon Application (6103) which the operator can consistentlyproduce. Using the example described above, these are the vowel soundsproduced by the operator of e as it sounds in green, i as it sounds inwhite, u as it sounds in blue, e as it sounds in red, o as it sounds inorange and e it sounds in grey. Each of these sounds is associatedrespectively with an identifier, for example, a number or a sequence ofone or more characters. The speech recognition access program (6105) ispreferably a Windows® application program developed using, in part, theVoice Tools Software Development Kit available from Dragon Systems, Inc.At initialization time, the speech recognition access program (6105)defines an array of data structures defining the menu and submenuoptions and the menu hierarchy. For example, one of the elements of thisarray determines that, on selection, certain actions are to be taken,for example, inputting text to an application program, and that certainsubmenu options and related data are to be associated with certain childedit windows. Also at initialization time, the speech recognition accessprogram (6105) initializes callback procedures using the DragonApplication's Application Program Interface to receive notification fromthe Dragon Application when a sound has been matched. Also atinitialization time, the main window of the speech recognition accessprogram (6105) is preferably sized to just encompass the soundindicators shown in FIG. 58. Also at initialization time, the speechrecognition access program (6105) creates six child edit controls, eachcorresponding to one of the sound indicators shown in FIG. 58. Each ofthe child edit controls has the background color described above and atext color of black or white depending upon which provides bettercontrast against the background color of the child edit control, andeach is located on the display (2112) as shown in FIG. 58. Also atinitialization time, the menu options of the initial menu are eachassociated respectively with one of the child edit controls. Afterinitialization, the speech recognition access program (6105), uponnotification from the Dragon Application that a sound has been receivedand a match attempted, sequentially searches the list of identifiersmatched to the sound by the Dragon Application (6103), starting with thebest match, until it finds an identifier corresponding to any one of thesix sounds distinguishable by the Dragon Application (6103). Assumingthe operator says o as it sounds in orange, the Dragon Application(6103) provides to the speech recognition access program (6105) a listof matches including, before any other identifier corresponding to anyone of the six sounds distinguishable by the Dragon Application (6103),the identifier associated with child edit control (5809). This childedit control is currently associated with menu option "vort x", thematched menu option. The speech recognition access program (6105) thensets the text of each of the child edit controls to one of the submenuoptions associated with the matched menu option. In the example above,submenu options space, "x", "r", "t", "v" and "o" are associated witheach of the child edit controls respectively and the text of theassociated child edit control is set to the submenu option. Assuming theoperator says e as in green, submenu option space is the matched submenuoption and a space is input to the application program (6107) whoseoutput appears in region (5850). Preferably the application program(6107) is an application program capable of executing a WM₋₋ SIZEcommand so that the speech recognition access program (6105) may sizethe windows of the application program (6107) to fit neatly in region(5850) and is capable of executing WM₋₋ CHAR messages so that the speechrecognition access program (6105) may input characters to theapplication program (6107).

J. Ideographic Language

The preferred embodiment of the Ideographic Language aspect of theinvention will now be described in detail from a functional perspectiveusing an example depicted in FIG. 64. FIG. 64 shows the display (2112)of a general purpose computer system (2218 in FIG. 15) and 12 selectableregions. Each of the 12 selectable regions consists of the union of avisible subregion on the display (2112) and an invisible subregionlocated outside the display (2112) and adjacent the visible subregion.For example, the selectable region in the upper left corner both aboveand below the top of the display (2112) in FIG. 64 consists of invisiblesubregion (3604) and visible subregion (3606), and hereinafter isreferred to as selectable region (3604/3606). The other selectableregions shown in FIG. 64, proceeding counter clockwise from selectableregion (3604/3606) are (3608/36,10), (3612/3614), (3616/3618),(3620/3622), (3624/3626), (3644/3646), (3648/3650), (3652/3654),(3656/3658), (3660/3662) and (3664/3666). In FIG. 64 each of the visiblesubregions is adjacent an edge of the display (2112). The selectableregions together circumscribe region (3680) in the center of thedisplay. Also shown on the display (2112) within region (3680) in FIG.64 are ten square indicating regions arranged in two columns of fiveindicating regions. Each indicating region abuts an indicating region inthe other column, the indicating region located above it, if any, andthe indicating region located below it, if any. In the preferredembodiment, each indicating region indicates by its location thelocation of a respectively associated selectable region, in accord withthe Location Indication aspect of the invention. For example, theuppermost indicating region in the left column of indicating regions(3605) is associated with the uppermost selectable region (3604/3606) onthe left side of the display. Indicating region (3609) locatedimmediately below indicating region (3605) is associated with selectableregion (3608/3610) located immediately below selectable region(3604/3606).

The operation of the example of the preferred embodiment of theIdeographic Language aspect of the invention will now be described.Assuming that the operator is fitted with a head pointer coupled to thegeneral purpose computer system (2218), that he uses a keyboard to entera phonetic unit and intonation according to the Pin Yin coding methodfor the Chinese language, and that he enters the distinct sound "fu",each of the plurality of the selectable regions shown in FIG. 64 isassociated with one of a plurality of menu options, each of theplurality of menu options having a common characteristic. In thisexample, the common characteristic is a phonetic unit and intonation butmay be a phonetic unit alone, an intonation alone, a stroke used to drawan ideograph, a number of horizontal strokes, a number of verticalstrokes, a number of total strokes, a stroke order, a radical, a part ofspeech, an ideograph, a kana, a diacritic, a classification of a part ofan ideograph or other characteristic of a class of ideographs. The tenmenu options shown in FIG. 64 are Chinese ideographs each starting withthe distinct sound "fu". Alternatively, the menu options may besequences of graphic symbols including one or more kanji. In FIG. 64 theChinese ideographs are each displayed on the visible subregion of theassociated selectable region and on the indicating region associatedwith the selectable region. The remaining two of the 12 selectableregions, (3624/3626) and (3644/3646), are associated with menu optionsfor undo and for displaying more menu options, respectively. In thepreferred embodiment, in response to the selection of the menu optionfor displaying more menu options, each of the selectable regionsassociated with a menu option starting with the distinct sound "fu" isassociated with a menu option not previously displayed and the newlyassociated menu option replaces the old menu option on the display.

Resuming now with the description of the example of the preferredembodiment, a cross hair cursor (3686) is displayed in the circumscribedregion (3680). Assuming the operator desires to select the menu optionassociated with selectable region (3608/3610), he turns his head to theleft and the cross hair cursor (3686) moves to the left, responsive tothe head movement, until the cross hair cursor hotspot intersectsselectable region (3608/3610) and he maintains the location of the crosshair cursor hotspot on that selectable region for the selectionthreshold period. The menu option associated with selectable region(3608/3610) is selected and added to text (3684) displayed in thecircumscribed region (3680), the general purpose computer system (2218)emits an audible beep indicating that selection has occurred and thedisplayed menu options, both on the indicating regions and the visiblesubregions, are removed from the display.

In the preferred embodiment, selection is made in accord with theFacilitated Dwell subaspect of the Dwell aspect of the invention,described above. Thus, the operator receives an indication of theprogress of his selection by a change in appearance of the indicatingregion associated with the intersected selectable region. Alternatively,selection may be by intersection of a location indicated by the at leastpart of a cursor and a selectable region alone, by such an intersectionaccompanied by a switch operation, for example, a depression of a spacebar on the keyboard, or by other suitable means.

In accord with the Location Indication aspect of the invention, theoperator sees the entire menu in the compact indicating regions and maydiscover the location of the selectable region associated with each menuoption without having to visually scan all the visible subregions. Inthe preferred embodiment, the plurality of indicating regions may bemoved to a different location on the display to avoid obstructing thearea of the display showing most recently added graphic symbols or thearea of the display where graphic symbols will soon be added. In accordwith the Ideographic Language aspect of the invention, an operator mayselect from among many sequences of one or more ideographs withoutlifting either hand from the keyboard, thus speeding entry of singleideographs or sequences of ideographs in word processing systems for theChinese, Japanese and Korean languages. Since, in the preferredembodiment, the selectable regions are located adjacent the edge of thedisplay, a large rectangular region remains available on the display forthe output of an application program. Further, if the selectable regionsare located entirely outside the display, the indicating regionsobstruct only a relatively small portion of the circumscribed region(3680), permitting the display of a sequences of ideographs forselection simultaneous with the display of previously selectedideographs, neither obstructing the operator's view of the other. If thegeneral purpose computer system is coupled to a speech synthesizer andthe ideographs are symbols of a symbol set, for example, theBlissymbolics Symbol Set, an illiterate operator, for example, a schoolchild having impaired speech, may select symbols associated with wordsand those words may be spoken via the speech synthesizer.

FIG. 65 illustrates a display and structures in accordance with analternative embodiment of the Ideographic Language aspect of theinvention. In this Figure, 18 selectable regions (3704/3706),(3708/3710), (3712/3714), (3716/3718), (3720/3722), (3724/3726),(3728/3730), (3732/3734), (3736/3738), (3740/3742), (3744/3746),(3748/3750), (3752/3754), (3756/3758), (3760/3762), (3764/3766),(3768/3770), (3772/3774) and (3776/3778) circumscribe region (3780) onthe display (2112). The visible subregions of four of the 18 selectableregions abut each of the top, left and right edges of the display(2112). The visible subregions of six of the 18 selectable regions abutthe bottom edge of the display (2112). The 18 indicating regions arelocated within the top half of region (3780). 12 of the 18 indicatingregions are arranged in two columns of six indicating regions each. Thecolumn on the left is close to the visible subregions abutting the leftedge of the display. The column on the right is close to the visiblesubregions abutting the right edge of the display. Two of the 18indicating regions are located between the top indicating regions ineach of the left and right columns. The remaining four of the 18indicating regions are located between the bottom indicating regions ineach of the left and right columns. Each of the four indicating regionsin the top row of indicating regions indicates the location of arespectively associated selectable region along the top edge of thedisplay. Each of the six indicating regions in the bottom row ofindicating regions indicates the location of a respectively associatedselectable region along the bottom edge of the display. Each of themiddle four indicating regions in each of the left and right columns ofindicating regions indicates the location of a respectively associatedselectable region along the left and right edge of the display. The 16menu options shown in FIG. 65 are Chinese ideographs each of whichincludes the radical kou, a radical having the shape of a rectangle.Each ideograph is displayed on the visible subregion of the associatedselectable region and on the indicating region associated with theselectable region. The remaining two of the 18 selectable regions,(3724/3726) and (3744/3746), are associated with menu options for undoand for displaying more menu options, respectively.

The indicating regions (3782) in FIG. 65 are disproportionately largerelative to the rest of FIG. 65. They are approximately 1.5 times theirproportionate size. They are represented as shown in FIG. 65 incompliance with Patent Cooperation Treaty Rules requiring a minimum sizefor letters in figures.

FIG. 66 illustrates a display and structures in accordance with analternative embodiment of the Ideographic Language aspect of theinvention. FIG. 66 depicts a display (2112) of a computer system (2116)having thereon ten square indicating regions, (3801), (3303), (3805),(3807), (3809), (3811), (3813), (3815), (3817), and (3819), arranged intwo columns of five indicating regions. Each indicating region abuts anindicating region in the other column, the indicating region locatedabove it, if any, and the indicating region located below it, if any.Each indicating region indicates the location of a respectivelyassociated selectable region (not shown) outside the display. Assumethat the operator is entering Chinese and uses a keyboard coupled to thecomputer system (2116) to enter the distinct sound "fu" . On each of theindicating regions is then displayed an ideograph having with thedistinct sound "fu". The operator, using a pointer coupled to thecomputer system (2116), points to the selectable region associated withthe indicating region on which is displayed the desired ideograph. Theselectable region is selected in response to a selection event, and, inthis example, the associated menu option is input to a word processingprogram and appears on the display (2112).

FIG. 67 illustrates a display and structures in accordance with analternative embodiment of the Ideographic Language aspect of theinvention. FIG. 67 depicts a display (2112) of a computer system (2116)having thereon eight selectable regions (6701), (6703), (6705), (6707),(6709), (6711), (6713), and (6715), each located on the display (2112)adjacent the edge of the display (2112) and associated respectively witha menu option. Together the eight selectable region circumscribe region(6750) on the display. In FIG. 67 each of eight menu options isdisplayed on its associated selectable region. Six of the eight menuoptions, associated with selectable regions (6703), (6705), (6707),(6701), (6715), and (6713) each represent a sequence of graphic symbolsin the Japanese language. These sequences, in Romanized Japanese arerespectively "hoka", "hoka ni", "hoka no", "hoka kara", "nanika hokanomono", and "hoka demo nai ga". Each sequence includes the kanji "hoka"shown alone on selectable region (6703). The sequences are ordered bylength, the shorter sequences on selectable regions on the left side ofthe display (2112) ordered by length from the top to the bottom of thedisplay (2112), the longer sequences along the right side of the display(2112) ordered by length from the top to the bottom of the display(2112). The remaining two of the eight selectable regions, (6709) and(6711), are associated with menu options for displaying the previous andthe next display of menu options, respectively. The menu optionassociated with a selectable region may be selected by a selectionevent.

The preferred embodiment of the Ideographic Language aspect of theinvention will now be described in detail from an implementationperspective. Preferably, the Ideographic Language aspect of theinvention is implemented by modifications to the access program (1206)described in the detailed description of the Perimeter Menu aspect ofthe invention, modified as described in the descriptions of theFacilitated Dwell subaspect and the Location Indication aspects of theinvention. The modifications required for the Ideographic Languageaspect of the invention are: (1) install on the general purpose computersystem a font for the ideographic language of the embodiment; (2) atinitialization time: (a) set the text of all labels for display on theselectable regions to null; (b) do not enable the cursor polling timer;and (c) hide the Windows® (1204) system cursor; (3) include in the mainwindow processing procedure of the access program (1206) code to processWM₋₋ CHAR messages and, when a sequence of one or more WM₋₋ CHARmessages indicates a common characteristic: (a) lookup sequences orrepresentations of sequences of one or more ideographs (hereinafter"sequences") having the common characteristic; (b) copy ten of thesequences to the labels; (c) set the Windows® (1204) system cursor tothe cross hair cursor, set the cursor location to a predeterminedlocation near an indicating region and show the Windows® (1204) systemcursor; (d) set fpaint to TRUE for every state machine; (e) send EV₋₋RESET to every state machine; (f) display each sequence at theappropriate location in the indicating region; and (g) enable the cursorpolling timer; and (4) add state processing to ST₋₋ SELECTED to: (a) setthe text of all labels for display to null; (b) hide the Windows® (1204)system cursor; (c) erase all sequences from the indicating region; (d)disable the cursor polling timer; (e) set fpaint to TRUE for every statemachine; (f) send EV₋₋ RESET to every state machine; and (g) insert theselected sequence into the work space.

In the above descriptions, there is shown and described only thepreferred and certain alternate embodiments of each aspect of theinvention, but, as aforementioned, it is to be understood that eachaspect of the invention is capable of use in various other combinationsand environments and is capable of changes or modifications within thescope of the inventive concepts as expressed herein.

    ______________________________________                                        Appendix I                                                                    ______________________________________                                        Procedure: PocketFsm                                                          /***********************************************************/                 /*                                     */                                     /*  Procedure: PocketFsm               */                                     /*                                     */                                     /*  Description:                                                                             State Machine for processing events to                                                                */                                     /*             selectable regions.     */                                     /*                                 */                                         /*  Input Parameters:              */                                         /*         pPocket  pointer to selectable region data                                                                */                                     /*                  set                */                                     /*         Event    event to process   */                                     /*                                     */                                     /***********************************************************/                 PocketFsm (pPocket, Event)                                                     BOOL   fInternalEvent                                                                             /* declaration of local                                                       boolean variable */                                       set fInternalEvent to TRUE                                                    while (fInternalEvent is TRUE)                                                {                                                                              set fInternalEvent to FALSE                                                   set pPocket->PreviousState to pPocket->State                                  set pPocket->State to aPocketFsm[pPocket->State][Event]                       switch (pPocket->State)                                                       {                                                                              case ST.sub.-- ILLEGAL.sub.-- STATE:                                          case ST.sub.-- ERROR:                                                          turn off all timers                                                           log the error                                                                 display an error message to the operator                                      break                                                                        case ST.sub.-- INITIAL:                                                        break                                                                        case ST.sub.-- RESET:                                                          set time of selection to the current time                                     if pPocket->Color is not equal to                                             pPocket- >InitialColor                                                         set pPocket->Color to pPocket->InitialColor                                   set pPocket->fPaint to TRUE                                                  set pPocket->fInvert to FALSE                                                 break                                                                        case ST.sub.-- LOW.sub.-- TIDE:                                                break                                                                        case ST.sub.-- CREST.sub.-- TIDE:                                              break                                                                        case ST.sub.-- BEGIN.sub.-- LOCK:                                              emit an audible beep                                                          set the system cursor location to the visible                                 subregion's lockspot                                                          set the system cursor to null                                                 display the lock icon on the visible subregion's                              lockspot                                                                                 /* initialize counter for                                                     locking */                                                         set iLockCursor to cLockCursor                                                break                                                                        case ST.sub.-- LOCK:                                                           set the system cursor location to the visible                                 subregion's lockspot                                                                     /* on lock counter                                                            expiration, */                                                                /* transfer to the next                                                       state */                                                           decrement iLockCursor by 1                                                    if iLockCursor equals 0                                                        set Event to EV.sub.-- NULL                                                   set fInternalEvent to TRUE                                                   break                                                                        case ST.sub.-- END.sub.-- LOCK:                                                erase the lock icon with white                                                set the system cursor to the arrow cursor                                     display the system cursor on the visible subregion'                           lockspot                                                                      emit an audible beep                                                                     /* paint selectable region so                                                 that **/                                                                      /* the white erasure of the                                                   lock */                                                                       /* cursor icon is                                                             overwritten */                                                     set pPocket->fPaint to TRUE                                                   break                                                                        case ST.sub.-- SELECTED:                                                       emit an audible beep                                                          take the action appropriate upon selection of this                            selectable region, for example, sending data to an                            application program, sending a control sequence to                            a device coupled to the computer, or displaying the                           selection                                                                     set pPocket->fInvert to TRUE                                                  if appropriate, change the menu options associated                            with various selectable regions and set fPaint to                             TRUE for the state machines associated with those                             selectable regions                                                            set Event to EV.sub.-- NULL                                                   set fInternalEvent to TRUE                                                    break                                                                        case ST.sub.-- SELECT.sub.-- AND.sub.-- OUT:                                   send EV.sub.-- RESET then EV.sub.-- MOVEMENT to all state                     machines, resetting them and indicating that the                              operator is moving the cursor                                                 break                                                                        case ST DECAY:                                                                            /* set state to previous                                                      state */                                                                      /* in preparation for the                                                     next */                                                                       /* state transition */                                             set pPocket->State to pPocket->PreviousState                                  decrement pPocket->Color by pPocket->Decrement, but                           not below pPocket->InitialColor                                               if pPocket->Color was changed                                                  set pPocket->fPaint to TRUE                                                   set fInternalEvent to TRUE                                                   if pPocket->Color was changed from a value greater                            than or equal to pPocket->CrestTide to a value below                          pPocket->CrestTide                                                             set Event to EV.sub.-- STEP.sub.-- DOWN                                      else                                                                           set Event to EV.sub.-- NULL                                                  break                                                                        case ST.sub.-- DWELL:                                                                     /* set state to previous                                                      state */                                                                      /* in preparation for the                                                     next */                                                                       /* state transition */                                             set pPocket->State to pPocket->PreviousState                                  increment pPocket->Color by pPocket->Increment, but                           not above pPocket->Ceiling                                                    if pPocket->Color was changed                                                  set pPocket->fPaint to TRUE                                                   set fInternalEvent to TRUE                                                   if pPocket->Color was changed from a value below                              pPocket->CrestTide to a value greater than or equal                           to pPocket->CrestTide                                                          set Event to EV.sub.-- STEP.sub.-- UP                                        else                                                                           set Event to EV.sub.-- NULL                                                  break                                                                        case ST.sub.-- IDLE:                                                           set the system cursor location to the center of the                           display                                                                       set fInternalEvent to TRUE                                                    set Event to EV.sub.-- NULL                                                   break                                                                        case ST.sub.-- EBB.sub.-- TIDE:                                                break                                                                        case ST.sub.-- ENTRY:                                                          set fInternalEvent to TRUE                                                    set Event to EV.sub.-- NULL                                                   break                                                                        case ST.sub.-- DISCARD:                                                                   /* set state to previous state */                                             /* for the next transition */                                      set pPocket->State to pPocket->PreviousState                                  break                                                                        case ST.sub.-- EXIT:                                                           set Event to EV.sub.-- NULL                                                   set fInternalEvent to TRUE                                                    if pPocket->fInterior equals TRUE                                              change the boundaries of the Windows ® region                             having the handle pPocket->hRegion to those                                   defined by the second array of points associated                              with this state machine                                                      set pPocket->fPaint to TRUE                                                   send EV.sub.-- RESET to the state machine having the index                    pPocket ->iAdjacentPocket                                                     break                                                                        default:                                                                       display error message                                                         break                                                                       }                                                                             /* end switch */                                                             }                                                                             /* end while */                                                               if pPocket->fPaint is equal to TRUE                                            invalidate the entire client area                                           }                                                                             /* end PocketFsm */                                                           Procedure: CreateEvent                                                        /***********************************************************/                 /*                                     */                                     /*  Procedure:                                                                              CreateEvent              */                                     /*                                     */                                     /*  Description:                                                                            Examines movement of cursor and determines                                                             */                                     /*            what events have occurred.                                                                             */                                     /*                                     */                                     /*  Input Parameters:              */                                         /*        pEvent  pointer to an event data structure                                                                 */                                     /*                for output           */                                     /*  Output:   A completed event data structure                                                                       */                                     /*            indicated by pEvent including:                                                                         */                                     /*            (1) an indication of whether the cursor hotspot                                                        */                                     /*              intersects a selectable region                                                                       */                                     /*            (2) the index of the intersected selectable                                                            */                                     /*              region, if any         */                                     /*            (3) the event for the intersected selectable                                                           */                                     /*              region, if any         */                                     /*            (4) the event for all non-intersected                                                                  */                                     /*              selectable regions     */                                     /*                                     */                                     /*            Updated variables indicate:                                                                            */                                     /*            (1) the previous location of the cursor hotspot                                                        */                                     /*            (2) the approximate time of the sampling of                                                            */e                                    /*              previous location of the cursor hotspot                                                              */                                     /*                                     */                                     /***********************************************************/                 CreateEvent (pEvent)                                                          {                                                                              get current cursor hotspot location                                           if there has been cursor hotspot movement since the last cursor               location sampling                                                              set time of previous cursor movement to time latest received                  WM.sub.-- TIMER message was generated                                         if cursor hotspot has crossed out of a selectable region                       set event to EV.sub.-- CROSS.sub.-- OUT for the intersected                selectable                                                                       region and to EV.sub.-- MOVEMENT for all other selectable                     regions                                                                      else                                                                           if cursor hotspot intersects a selectable region                               if intersected selectable region's Color equals                               Ceiling                                                                        set event to EV.sub.-- CEILING for the intersected                            selectable region and to EV.sub.-- DECAY for all other                        selectable regions                                                           else                                                                           set event to EV.sub.-- DWELL for the intersected                              selectable region and to EV.sub.-- DECAY for all other                        selectable regions                                                          else                                                                           set event to EV.sub.-- MOVEMENT for all selectable regions                 else                                                                           if cursor hotspot intersects a selectable region                               if idle time exceeded                                                          set event to EV.sub.-- IDLE.sub.-- TIMEOUT for all selectable                 regions                                                                      else                                                                           if intersected selectable region's Color equals                               Ceiling                                                                        set event to EV.sub.-- CEILING for the intersected                            selectable region and to EV.sub.-- DECAY for all other                        selectable regions                                                           else                                                                           set event to EV.sub.-- DWELL for the intersected                              selectable region and to EV.sub.-- DECAY for all other                        selectable regions                                                         else                                                                           set event to EV.sub.-- DECAY for all selectable regions                     set the previous cursor hotspot location to the current hotspot               location                                                                     }                                                                             /* end CreateEvent */                                                         ______________________________________                                    

I claim:
 1. A method of selecting a sequence of one or more graphicsymbols from a first plurality of sequences of one or more graphicsymbols, one or more sequences of the first plurality of sequencesincluding one or more ideographs in an ideographic language, said methodcomprising the steps of:displaying on a display the first plurality ofsequences, each of the first plurality of sequences having a firstcommon characteristic in the ideographic language and each associatedrespectively with one of a first plurality of displayed soundindicators, each of the first sound indicators respectively indicatingone of a first plurality of indicated sounds, each of the firstindicated sounds differing from each other first indicated sound;matching a first received sound to any one of the first indicatedsounds; and selecting the sequence associated with the matched firstindicated sound.
 2. The method of claim 1 wherein the first receivedsound is not a phonetic representation of the sequence associated withthe matched first indicated sound.
 3. The method of claim 1 wherein oneof the first sound indicators is non-alphanumeric.
 4. The method ofclaim 3 wherein the non-alphanumeric sound indicator is a hue.
 5. Themethod of claim 3 wherein the non-alphanumeric indicator includes anyone of a pitch indicator, a volume indicator, a sound durationindicator, a change in pitch indicator, a change in volume indicator, ashape indicator, a color saturation indicator, a brightness indicator, alocation indicator, and an object indicator.
 6. The method of claim 1further comprising the step of indicating the association of each of thefirst plurality of sequences with the associated sound indicator.
 7. Themethod of claim 1 wherein each of the first plurality of sequencesbelongs to any one of the Chinese language, Japanese language, Koreanlanguage, Picture Communication Symbols symbol set, Rebus symbol set,Picsyms symbol set, Pictogram Ideogram Communication Symbols symbol set,Yerkish symbol set, Blissymbolics symbol set, and a set of depictions ofthe signs of a manual sign language.
 8. The method of claim 1 whereinthe first common characteristic includes a phonetic unit, includingeither one of a Chinese pronunciation and a Japanese pronunciation. 9.The method of claim 8 wherein the phonetic unit is specified by voice.10. The method of claim 8 wherein each of the first sound indicatorsincludes a hue of a region on the display each region having a huedifferent from the hue of all the regions of all other first soundindicators; wherein each region is either adjacent or nearly adjacentanother of the regions; wherein the displaying step displays eachsequence intersecting the region of the associated sound indicator; andwherein the first received sound is the name of the hue of the soundindicator to be matched.
 11. The method of claim 10 further comprisingthe step of inputting the selected sequence to an application program.12. The method of claim 8 wherein the phonetic unit is specified by oneor more keystrokes.
 13. The method of claim 8 wherein the phonetic unitis a phonetic unit in a first dialect of the ideographic language, andwherein the matching step includes matching the phonetic unit to aphonetic unit in a second dialect of the ideographic language.
 14. Themethod of claim 1 wherein the first common characteristic includes anyone of:(a) an intonation; (b) a distinct sound; (c) a part of speech;(d) a meaning; and (e) a meaning class.
 15. The method of claim 1wherein the first common characteristic includes any one of:(a) a strokeused to draw an ideograph; (b) a number of horizontal strokes; (c) anumber of vertical strokes; (d) a number of total strokes; (e) a strokeorder; (f) a radical; (g) an ideograph; (h) a kana; (i) a diacritic; (j)a classification of a part of an ideograph; (k) a root; (l) a graphicsymbol; (m) an appearance of a graphic symbol; and (n) an attribute foroptical recognition purposes.
 16. The method of claim 1 wherein thedisplaying step precedes the matching step.
 17. The method of claim 1wherein the first plurality of sound indicators together at leastpartially circumscribe a region on the display.
 18. The method of claim17 wherein each of the first sound indicators is outside the display.19. The method of claim 1 wherein each of the first sound indicators ison the display and either adjacent or nearly adjacent another of thefirst sound indicators.
 20. The method of claim 1 further comprising thesteps of:receiving a signal; and responsive to the received signal,displaying on the display a second plurality of sequences of one or moregraphic symbols, each of the second plurality of sequences having thefirst common characteristic and each associated respectively with one ofa second plurality of displayed sound indicators, each of the secondsound indicators respectively indicating one of a second plurality ofindicated sounds, each of the second indicated sounds differing fromeach other second indicated sound.
 21. The method of claim 20 whereineach sequence of the first plurality of sequences has a frequency of usegreater than each sequence of the second plurality of sequences.
 22. Themethod of claim 1 further comprising the steps of:first storing arepresentation of the first common characteristics; and second storing arepresentation of a second common characteristic.
 23. The method ofclaim 22 wherein the first storing step and the second storing stepprecede the displaying step.
 24. The method of claim 23 followed by thesteps of:displaying oil the display a second plurality of sequences ofone or more graphic symbols, each of the second plurality of sequenceshaving the second common characteristic and each associated respectivelywith one of a second plurality of displayed sound indicators, each ofthe second sound indicators indicating one of a second plurality ofindicated sounds, each of the second indicated sounds differing fromeach other second indicated sound; matching a second received sound toany one of the second indicated sounds; and selecting the sequence ofthe second plurality of sequences associated with the matched secondindicated sound.
 25. The method of claim 1 wherein one of the firstsound indicators is numeric.
 26. The method of claim 1 wherein thedisplaying step displays each sequence of the first plurality ofsequences in an order responsive to a second characteristic of thesequence other than the first common characteristic, the secondcharacteristic including any one of:(a) a phonetic unit, includingeither one of a Chinese pronunciation and a Japanese pronunciation; (b)an intonation; (c) a distinct sound; (d) a stroke used to draw anideograph; (e) a number of horizontal strokes; (f) a number of verticalstrokes; (g) a number of total strokes; (h) a stroke order; (i) aradical; (j) a part of speech; (k) an ideograph; (l) a kana; (m) adiacritic; (n) a classification of a part of an ideograph; (o) ameaning; (p) a meaning class; (q) a root; (r) a graphic symbol; (s) anappearance of a graphic symbol; (t) an attribute for optical recognitionpurposes; (u) a frequency of use of each of the first plurality ofsequences; (v) a number of graphic symbols; and (w) a frequency of useof each of a second plurality of sequences, of graphic symbols, each ofthe second plurality of sequences including one of the first pluralityof sequences.
 27. The method of claim 1 further comprising the step ofinputting the selected sequence to an application program.
 28. Themethod of claim 1 further comprising the steps of:inputting a firstsequence of one or more graphic symbols including one or more ideographsto an application program; and replacing the first sequence with theselected sequence.
 29. The method of claim 1 wherein the first commoncharacteristic includes a first graphic symbol and the displaying stepdisplays only a single instance of the first graphic symbol indicatingthat the first graphic symbol is included in each of the first pluralityof sequences.
 30. The method of claim 29 wherein the first graphicsymbol is an ideograph.
 31. The method of claim 29 wherein thedisplaying step displays the single instance of the first graphic symbolin a region of the display separate from the display of at least part ofone of the first plurality of sequences or in a manner visibly distinctfrom the display of at least part of one of the first plurality ofsequences.
 32. The method of claim 1 wherein one of the first soundindicators includes at least part of a particular graphic symbol of theassociated sequence.
 33. The method of claim 32 wherein the firstreceived sound includes any one of:(a) a pronunciation of at least partof the particular graphic symbol; (b) a name of at least part of theparticular graphic symbol; (c) a meaning of at least part of theparticular graphic symbol; (d) a meaning class of at least part of theparticular graphic symbol; (e) a part of speech of at least part of theparticular graphic symbol; (f) a classification of at least part of theparticular graphic symbol; (g) a stroke used to draw at least part ofthe particular graphic symbol; (h) a number of horizontal strokes usedto draw the particular graphic symbol; (i) a number of vertical strokesused to draw the particular graphic symbol; and (j) a number of totalstrokes used to draw the particular graphic symbol.
 34. The method ofclaim 1 wherein the number of sequences in the first plurality ofsequences is less than or equal to a predetermined number therebyincreasing the likelihood of matching of the first received sound to thefirst indicated sound intended by a user.
 35. For use in a language richin homophones, a method of selecting a sequence of one or more graphicsymbols in the language from a plurality of sequences of one or moregraphic symbols in the language, said method comprising the steps of:(a)displaying the plurality of sequences, each of the plurality ofsequences associated respectively with a sound which is not a phoneticrepresentation of the associated sequence, each of the plurality ofsounds differing from each other sound in the plurality of sounds; (b)receiving a sound signal; (c) matching the received sound signal to anyone of the plurality of sounds; and (d) selecting the sequenceassociated with the matched sound.
 36. The method of claim 35 wherein atleast two of the sequences are homophones.
 37. The method of claim 35wherein each sequence represents a word in the language.
 38. The methodof claim 35 further comprising the steps of:(e) displaying a pluralityof sound indicators, each sound indicator associated respectively withone of the plurality of sounds; and (f) indicating the association ofeach sequence to the sound indicator associated with the soundassociated with the sequence.
 39. A method of editing a first sequenceof two or more words in an ideographic language, said method comprisingthe steps of:displaying on a display:(a) the first sequence, the firstsequence including a plurality of second sequences of one or moregraphic symbols in the language, each of the second sequences associatedrespectively with a sound indicator, each of the sound indicatorsrespectively indicating a sound, each of the indicated sounds differingfrom each of the other indicated sounds; and (b) the sound indicators;receiving a sound; matching the received sound to any one of theindicated sounds; and replacing the second sequence associated with thematched indicated sound with a third sequence of one or more graphicsymbols in the language, the third sequence including an ideograph inthe language.
 40. For use with a general purpose computer systemincluding a display, an apparatus for selecting a sequence of one ormore graphic symbols from a plurality of sequences of one or moregraphic symbols, one or more sequences of the plurality of sequencesincluding an ideograph in an ideographic language, said apparatuscomprising:(a) a computer readable medium; and (b) a program, stored onthe medium and executable by the general purpose computer system,for:(1) displaying the plurality of sequences on the display, each ofthe sequences associated respectively with a displayed sound indicator,each of the sound indicators respectively indicating a sound, each ofthe plurality of indicated sounds differing from each of the otherindicated sounds, and wherein a particular one of the sound indicatorsis not a phonetic representation of the sequence associated with theparticular sound indicator; (2) matching a received sound to the soundindicated by the particular sound indicator; and (3) selecting thesequence associated with the particular sound indicator.
 41. Theapparatus of claim 40 wherein each of the plurality of sequences has acommon characteristic in the ideographic language.
 42. The apparatus ofclaim 40 wherein the medium may be coupled to and uncoupled from thegeneral purpose computer system.
 43. The apparatus of claim 40 whereinthe medium includes any one of:(a) a random access memory; (b) amagnetic store; and (c) an optical store.
 44. An apparatus for selectinga sequence of one or more graphic symbols from a plurality of sequencesof one or more graphic symbols, one or more sequences of the pluralityof sequences including one or more ideographs in an ideographiclanguage, said apparatus comprising:(a) a display on which may bedisplayed the plurality of sequences, each of the plurality of sequenceshaving a common characteristic in the ideographic language and eachassociated respectively with a displayed sound indicator, each of thesound indicators respectively indicating a sound, each of the pluralityof indicated sounds differing from each of the other indicated sounds,and wherein a particular one of the sound indicators is not a phoneticrepresentation of the sequence associated with the particular soundindicator; and (b) a selection device for matching a received sound tothe sound indicated by the particular sound indicator and for selectingthe sequence associated with the particular sound indicator.
 45. Theapparatus of claim 44 wherein the common characteristic includes any oneof the meaning classes:(a) actions; (b) amounts; (c) animals; (d)articles of clothing; (e) bodily functions; (f) buildings; (g) businessactivities; (h) cleaning activities; (i) colors; (j) communicationactivities; (k) computer peripherals; (l) days; (m) devices used tomaintain personal hygiene; (n) directions; (o) drinks; (p) emergencyconditions; (q) emotions; (r) financial activities; (s) foods; (t)government services; (u) greetings; (v) holidays; (w) householdappliances; (x) illnesses; (y) items of office equipment; (z) jokes;(aa) lengths; (ab) locations, including locations frequented by a userof the apparatus; (ac) meals; (ad) means of transportation; (ae) months;(af) names; (ag) numbers; (ah) parts of the human body; (ai) personsknown to a user of the apparatus; (aj) plants; (ak) prosthetic device;(al) recreational activities; (am) rehabilitation activities; (an)relative locations; (ao) school activities; (ap) shapes; (aq) shoppingactivities; (ar) sizes; (as) smells; (at) sports; (au) tactileattributes; (av) tastes; (aw) telephone numbers; (ax) temperatures; (ay)times; (az) topics of study; (ba) utterances used as acknowledgements inconversation without conveying new substantive information; (bb)utterances used to bid for a turn to speak in conversation; (bc)weights; and (bd) work activities.
 46. The apparatus of claim 44 furthercomprising:(a) a computer system including the display and a soundreceiver for receiving the received sound; and (b) a program, executableon the computer system; and wherein the selection device is formed bythe combination of the progam and the computer system.
 47. The apparatusof claim 44 wherein the selection device includes a processor, a programexecutable oil the processor, and a medium readable by the processor;wherein the processor, during execution of the program, compares arepresentation of the received sound to a representation of each of theindicated sounds, the representation of each of the indicated soundsstored on the medium and read by the processor therefrom; and whereinthe sound indicated by tile particular sound indicator is tile indicatedsound whose representation is closest to the representation of thereceived sound.
 48. The apparatus of claim 44 wherein the selectiondevice includes a speech recognizer for comparing the received sound tothe sound indicated by the particular sound indicator.
 49. The apparatusof claim 40 wherein the program is transferred from the medium to thegeneral purpose computer system via a telecommunications or datacommunications network.
 50. The apparatus of claim 44 wherein the soundindicated by the particular sound indicator is either one of:(a) a soundfound in normal speech; and (b) a sound not found in normal speech. 51.The apparatus of claim 44 wherein each of the plurality of sequencesbelongs to any one of the Chinese language, Japanese language, Koreanlanguage, Picture Communication Symbols symbol set, Rebus symbol set,Picsyms symbol set, Pictogram Ideogram Communication Symbols symbol set,Yerkish symbol set, Blissymbolics symbol set, and a set of depictions ofthe signs of a manual sign language.
 52. The apparatus of claim 44wherein the plurality of sound indicators together at least partiallycircumscribe a region on the display.
 53. The apparatus of claim 52wherein each of the sound indicators is outside the display.
 54. Anapparatus for selecting a sequence of one or more graphic symbols from aplurality of sequences of one or more graphic symbols, one or moresequences of tile plurality of sequences including one or moreideographs in an ideographic language, said apparatus comprising:(a)means for displaying tile plurality of sequences, each of tile pluralityof sequences having a common characteristic in the ideographic languageand each associated respectively with a displayed sound indicator, eachof the sound indicators respectively indicating a sound, each of theplurality of indicated sounds differing from each of the other indicatedsounds, and wherein a particular one of the sound indicators is not aphonetic representation of the sequence associated with tile particularsound indicator: and (b) means for:(1) matching a sound to the soundindicated by the particular sound indicator; and (2) selecting thesequence associated with the particular sound indicator.
 55. A devicecontroller comprising:(a) a display on which may be displayed aplurality of sequences of one or more graphic symbols, one or moresequences of the plurality of sequences including one or more ideographsin an ideographic language, each of the plurality of sequences having acommon characteristic in the ideographic language and each associatedrespectively with a displayed sound indicator, each of the soundindicators respectively indicating a sound, each of the plurality ofindicated sounds differing from each of the other indicated sounds, andwherein a particular one of the sound indicators is not a phoneticrepresentation of the sequence associated with the particular soundindicator; and (b) a signal generating device, coupled to a controlleddevice, for:(1) matching a sound to the sound indicated by theparticular sound indicator, the sequence associated with the particularsound indicator representing a function of the controlled device; and(2) generating a device control signal corresponding to the function ofthe controlled device.
 56. The apparatus of claim 55 wherein thecontrolled device includes any one of:(a) a computer peripheral; (b) adevice capable of playing previously recorded sound; (c) a devicecapable of playing previously recorded video; (d) a household appliance;(e) a lamp; (f) a microprocessor; (g) a motorized transport deviceincluding either one of a scooter and a wheelchair; (h) a radio; (i) arobot; (j) a security system; (k) a television; (l) a thermostat; (m) avoice output device; (n) a workstation; (o) an alarm; and (p) an officeappliance.